Interviewer calibration guide authoring

Deep reference for qa-hiring-kit SKILL.md - producing the calibration guide, the what each score looks like leg of the structured-interview tripod.

Why a calibration guide

A rubric describes how to score; a calibration guide demonstrates what a score looks like. The structured-interview research is consistent that anchor descriptions reduce inter-rater noise but do not eliminate it - interviewers still drift on edge cases without worked examples of the same question scored across the whole 1-4 range.

The calibration guide is the third leg of the structured-interview tripod:

• interview-questions.md (opens in new window) - what we ask
• rubric.md (opens in new window) - how to score
• this guide - what each score looks like in practice

Without all three, an interview loop is structurally incomplete - the team will produce an inconsistent signal regardless of how good the questions and rubric are individually. The guide is panel-internal: sharing the gold-standard answers with candidates pollutes the signal.

Step 1 - Capture the inputs

If the team is launching a new role with no prior transcripts, use synthesised transcripts (emit transcripts that demonstrate each score level) but flag explicitly that the panel must replace them with real anonymised transcripts after the first 5 candidates.

Step 2 - Per-question gold-standard answers

For each question in the bank, the calibration guide emits four worked answers demonstrating each rubric level (1 = no hire, 2 = borderline, 3 = hire, 4 = strong hire):

### Q3 - Behavioral (STAR): late-defect catch - gold standards

#### Score 1 - no hire

> "Yeah, last release we caught a really bad bug right before launch. The whole team was upset. We were lucky we caught it. After that I'm just always more careful."

**Why score 1:** No technique named; no STAR structure (S/T/A/R undifferentiated); attributes the catch to luck and emotional response rather than process. Anchor: "Cannot articulate a partition / boundary / decision-table technique" - matches.

#### Score 2 - borderline

> "Last release I caught a defect where the cart total wrapped around at $99.99 to a negative number. I added a boundary test for the max-price limit. It was fine after that."

**Why score 2:** Names a technique (boundary value analysis, implicitly); STAR is partial - Situation and Action are clear; Task is implicit ("I was the QA lead"); Result is "it was fine after that" - no measurable outcome, no retrospective learning. Probe for the Task and Result; if the candidate fills both, score moves to 3.

**Probe to use:** "What was your specific role at that point? And how did the team know the fix worked beyond that one test?"

#### Score 3 - hire

> "On the v3.4.0 release we caught a defect where the cart total wrapped around at $99.99 to a negative number. As the QA lead for the release, I was running the standard regression suite when I noticed our test cases all used round-dollar amounts - $50, $100. I added boundary-value tests at the upper limit (one cent under, exactly at, one cent over) and the bug reproduced at exactly $99.99. We fixed the cents-precision rounding in the price-display component, added the boundary tests to the regression suite, and updated our test-data conventions to use Faker's `randomFloat({ precision: 0.01 })` rather than round numbers."

**Why score 3:** Specific technique (boundary value analysis) named **and** applied. STAR complete: situation (v3.4.0), task (QA lead, running regression), specific action (noticed pattern, added boundary tests), measurable result + retro learning (fix shipped, regression added, conventions updated).

#### Score 4 - strong hire

> "[Same situation as score 3, plus:] After we shipped the fix, I noticed our test-data conventions had no rule against round-number-only test data. I authored a one-page conventions update that mandated Faker boundary inputs for any numeric field with a documented constraint, and got the rest of the QA team to sign off. Over the following two releases, we caught two more rounding-precision bugs at the same boundary class - proving the convention change was load-bearing, not just paperwork."

**Why score 4:** Generalises beyond the specific defect to a systemic process change, ties the change to a measurable downstream outcome (two more bugs caught), and documents organisational influence (got the team to sign off).

The four-level demonstration is the load-bearing artifact. An interviewer who reads all four can grade ambiguous transcripts by asking which of these does this candidate's answer most resemble - concrete comparison rather than abstract scoring.

Step 3 - Common pitfalls

For each question, the guide emits a "common pitfalls" section - the typical mistakes interviewers make scoring this question:

### Q3 - common interviewer pitfalls

| Pitfall | Why it produces noise | Correction |
|---|---|---|
| Scoring on tone of voice / "confidence" | Tone is interviewer-specific; the same candidate scored differently by interviewer A and B on tone alone. | Score on what the candidate said, not how. |
| Awarding score 3 because the candidate "is clearly senior" | Halo effect; the question's anchor doesn't mention seniority. | Strict adherence to the anchor's behavioural description. |
| Probing too aggressively after a partial answer | Different interviewers probe to different depths; a thoroughly-probed candidate looks stronger than a lightly-probed one with the same competence. | Use only the pre-authored probes from the question bank; one probe per missing component, no more. |
| Stopping at score 3 because "I never give 4s" | Anchor calibration drift; the rubric's level 4 exists for a reason. | If the candidate matches the level-4 anchor, give level 4; review your prior interviews for under-scoring. |
| Awarding score 4 because "the answer was great" without checking the systemic-gap and measurable-outcome anchors | Halo effect on level 4. | Level 4 requires both anchor sub-conditions (systemic + measurable); level 3 is the default for excellent answers that don't cross both. |

Step 4 - Calibration-session script

The guide's last section is the script for the panel's calibration meeting - typically 90 minutes for a 6-question loop:

1. Pre-read (the panel reads the guide's gold-standards individually, ~30 minutes before the meeting).
2. Per-question discussion (~10 minutes per question): the panel scores the same anonymised real transcript independently, reveals scores, discusses any disagreement until the panel converges on a score.
3. Anchor refinement (~10 minutes): if the panel discovers the rubric's anchor is ambiguous, edit the rubric (and document the change). Calibration sessions are the canonical moment to refine the rubric.
4. Pitfall review (~10 minutes): the panel walks through Step 3's common pitfalls; each interviewer flags one they recognise from their own prior interviews.

The script is mandatory before the first real candidate. Per the structured-interview research, calibration is the dominant variable in inter-rater agreement - more important than rubric quality alone.

Step 5 - Emit the guide

The output is a single markdown document with:

1. Header: role, seniority, source question bank + rubric references, panel size, recommended session length.
2. Per-question gold standards (Step 2 - four worked answers per question).
3. Per-question pitfalls (Step 3 - common interviewer mistakes).
4. Calibration-session script (Step 4 - agenda + timing).
5. Anchor-drift log (initially empty; the panel records every rubric anchor that was tightened during the session, so future hiring rounds inherit the refinement).
6. Hand-off block: run the calibration session before any real candidate; after the first 5 real candidates, retro the questions and replace synthesised transcripts with real anonymised ones; re-author the guide whenever the role description, seniority, or rubric change.

Anti-patterns

Limitations

• Synthesised transcripts are educated guesses, not real data. They demonstrate the rubric's intent but do not reflect actual candidate variance. Replace ASAP.
• Calibration is bounded by panel discipline. A panel that nods through the script without genuinely scoring transcripts together will not achieve inter-rater agreement.
• The guide is panel-internal and should not leave the hiring loop. Storing it in the team's wiki is fine; sending it to candidates is not.
• Anchor refinement is a living artifact. The rubric and this guide co-evolve; the team's wiki should hold the latest version, and prior versions should be archived per hiring round.
• No fairness / bias audit. The gold-standard answers are not checked for class bias - that is the team's HR / legal review.

References

• Structured interview research - Levashina et al. 2014 (Personnel Psychology (opens in new window)) on the validity uplift from calibrated structured interviews; the methodological basis for the "gold-standards + calibration session" defaults.
• STAR behavioral interviewing method - Situation / Task / Action / Result framework, the structure of the gold-standard answers for behavioural questions: https://en.wikipedia.org/wiki/Situation,_task,_action,_result
• ISTQB glossary - the canonical QA vocabulary the gold-standard answers reference (defect, defect-density, escaped-defect, test-design-technique, etc.): https://glossary.istqb.org/
• Bloom's taxonomy - the K1 - K4 cognitive-difficulty levels used to ensure gold-standard answers match the question's intended depth: https://en.wikipedia.org/wiki/Bloom%27s_taxonomy
• PractiTest 2026 State of Testing Report - hiring rubric / interviewer calibration cited as a high-adoption, low-risk AI use case for QA managers: https://www.practitest.com/state-of-testing/
• interview-questions.md (opens in new window), rubric.md (opens in new window) - the two upstream references that produce the questions and the rubric this guide demonstrates.

Interview question bank authoring

Deep reference for qa-hiring-kit SKILL.md - producing the QA-role-specific question bank, the what we ask leg of the structured-interview tripod.

Why structured questions

Hiring for QA roles is calibration-heavy: the same question scored by two interviewers without a rubric produces high noise (well-documented in structured-interview research (opens in new window)). The remedy is a structured interview - same questions, same order, same scoring rubric across candidates. This reference produces the questions half of that pair; the scoring rubric (rubric.md (opens in new window)) and the calibration guide (calibration.md (opens in new window)) are the other legs.

The bank is QA-specific by design: (a) ISTQB-aligned competency framing, (b) role-specific question depth (manual QA vs SDET vs test lead require different technical / behavioral mixes), (c) STAR-format anchoring on behavioral questions per the canonical STAR method (opens in new window), and (d) a structured output ready to drop into a hiring loop.

Step 1 - Capture the role inputs

If the role title is not one of the five recognised QA roles, halt with UNRECOGNISED_ROLE: supply a role from the recognised list, or run with role=qa-generic to use a flat default mix.

Step 2 - Allocate question slots

A typical 60-minute interview holds 6 - 8 questions; default to a six-question shape. The mix shifts per role:

The mix is configurable; the table is the default. Behavioral count grows with seniority and people-leadership scope per the structured-interview research; technical depth grows with hands-on coding scope.

Step 3 - Author per-slot questions

For each slot, emit one question with the metadata reviewers need:

### Q3 - Behavioral (STAR) | Senior | Bloom: K3 (Apply)

**Question:** Tell me about a release where you caught a critical defect late - after the test cycle but before production. Walk me through the situation, what your role was, what you did, and what the team learned.

**ISTQB competency:** Defect management (defect → failure distinction, escape-defect lifecycle).
**STAR cues:** Listen for: (S) the release context, (T) the candidate's specific responsibility, (A) the diagnostic and communication actions taken, (R) measurable outcomes + retro learnings.
**Time budget:** 8 min.
**Follow-up probes** (use only if the answer is shallow):
- "Was the defect found through automated tests, manual exploration, or a customer report?"
- "What changed in your team's process after this incident?"
- "How did you handle the stakeholder communication?"

Each question carries:

• Type: technical / behavioral / scenario / system-design.
• Seniority anchor: which level the question is calibrated for; senior questions remain useful at staff+ but with deeper probes.
• Bloom's level (K1 - K4): K1 (remember) for fundamentals, K2 (understand) for explanation, K3 (apply) for situational execution, K4 (analyse) for synthesis. Bloom's levels are used by ISTQB to anchor question difficulty across the Foundation Level syllabus.
• ISTQB competency the question maps to (cite by stable ID; the ISTQB Foundation Level v4.0 syllabus (opens in new window) is the canonical competency reference, though its specific URL drift means cite by syllabus version).
• STAR cues (for behavioral questions only) - per the STAR method (opens in new window), the interviewer listens for all four components; missing components prompt follow-up probes.
• Time budget in minutes.
• Follow-up probes - pre-authored so different interviewers don't drift into ad-hoc probing (the dominant source of interview noise per structured-interview research).

Step 4 - Tune the difficulty distribution

Bloom's taxonomy mix per seniority (default; configurable):

Flag questions whose Bloom's level is too far from the role's centre of gravity (e.g., a K1 fundamental question for a staff+ candidate is a wasted slot).

Step 5 - Emit the bank

The output is a single markdown document with:

1. Header: role, seniority, domain, total slots, mix summary.
2. Question slots 1 - N: per the format in Step 3.
3. Reading list for the candidate (optional): canonical references the candidate can prepare against - ISTQB Foundation Level syllabus, the team's testing handbook, etc. Always optional; avoid if the role advertises "no prep needed".
4. Hand-off block: pair with rubric.md (opens in new window) for the per-question scoring rubric (the rubric and the questions must travel together; otherwise the loop reverts to unstructured), then calibration.md (opens in new window) once the rubric exists. Lock the question bank at the start of the hiring round; if the bank changes mid-round, every prior candidate's score is no longer comparable.

Anti-patterns

Limitations

• The bank is not a substitute for technical screening. A take-home or coding round is still recommended for SDET / automation roles where hands-on code is load-bearing. This produces interview questions, not coding exercises.
• STAR cues are heuristics, not auto-scoring. The interviewer still has to listen and score. The cue list reduces variance but does not eliminate it.
• Bloom's levels are the ISTQB-canonical proxy, not a perfect difficulty measure. A K3 question can be more or less difficult than another K3 depending on the topic. Use the level as a coarse mix knob.
• Locale / language localisation is the team's responsibility. Behavioral questions translated literally can lose nuance; the defaults are English-language.
• Legal compliance varies by jurisdiction. Some questions (age, family, health) are illegal in some jurisdictions and merely poor practice in others. The default forbidden-topics list covers age, family status, religion, sexual orientation, disability, national origin, and citizenship except where role-required, but the hiring team is responsible for jurisdiction-specific compliance.

References

• ISTQB Certified Tester Foundation Level v4.0 syllabus - competency areas (test design, test management, test process, defect management, tools); cite by stable syllabus version (v4.0, 2023): https://www.istqb.org/certifications/certified-tester-foundation-level
• ISTQB glossary - defect, defect-density, escaped-defect, test design technique (the canonical competency vocabulary): https://glossary.istqb.org/
• STAR behavioral interviewing method - Situation / Task / Action / Result framework: https://en.wikipedia.org/wiki/Situation,_task,_action,_result
• Structured interview research - Levashina et al. 2014 (Personnel Psychology (opens in new window)) on the validity uplift from structured employment interviews; the methodological basis for the "same questions, same order, same scoring" defaults.
• Bloom's taxonomy - K1 remember / K2 understand / K3 apply / K4 analyse - ISTQB-adopted cognitive-difficulty levels: https://en.wikipedia.org/wiki/Bloom%27s_taxonomy
• PractiTest 2026 State of Testing Report - hiring rubric / interview question generation cited as a high-adoption, low-risk AI use case for QA managers: https://www.practitest.com/state-of-testing/
• rubric.md (opens in new window), calibration.md (opens in new window) - the sibling references that complete the structured-interview triple.

Job description authoring

Deep reference for qa-hiring-kit SKILL.md - authoring the QA job description, the upstream-most artifact of the hiring chain.

Why the JD comes first

The JD is the first scoring instrument in the hiring chain, applied by candidates to themselves: a vague posting screens nobody, and an everything-list screens out exactly the people the team wants. The JD's responsibilities come from the role's actual test activities and its skills section uses the same competency vocabulary the downstream rubric (rubric.md (opens in new window)) will score - so the role a candidate applies to is the role the panel evaluates.

Two grounding sources. For what the role does: ISTQB CTFL v4.0 section 1.4.5 defines two principal roles in testing - the testing role, which "takes overall responsibility for the engineering (technical) aspect of testing" and is "mainly focused on the activities of test analysis, test design, test implementation and test execution", and the test management role, which "takes overall responsibility for the test process, test team and leadership of the test activities" and is "mainly focused on the activities of test planning, test monitoring and control and test completion". The same section notes one person may take on both roles at the same time (ISTQB CTFL Syllabus v4.0, §1.4.5 (opens in new window); syllabus text verified 2026-06-10 from the published PDF). For how the document reads: Workable's JD guidance - clear, standard job titles (creative titles like "Rockstar Engineer" read as "unrealistic and potentially discriminatory"), 300 - 660 words total, bulleted duties that show a typical workday, and requirements split into must-have versus nice-to-have (Workable, "How to write a good job description" (opens in new window), fetched 2026-06-10).

Step 1 - Capture the inputs

Step 2 - Derive responsibilities from the role's test activities

Write 5 - 8 responsibility bullets, each traceable to a test activity, phrased as a typical-workday action (Workable's guidance: duties should show what a normal day looks like, not an aspirational mission). Map per role balance:

• Testing-role bullets draw on test analysis, test design, test implementation, test execution (CTFL §1.4.5 (opens in new window)): "design and automate API-level regression tests for the payments services", "run and extend exploratory charters on new checkout features".
• Test-management bullets draw on test planning, monitoring and control, completion: "own the test strategy section for your product area", "report quality status to engineering leadership each sprint".

A role that is 100% one kind needs no bullets from the other; a mixed role states the split rather than hiding it ("~70% hands-on automation, ~30% process ownership").

Step 3 - Split skills into must-have vs nice-to-have

The split is the JD's main screening mechanism (Workable: be upfront about non-negotiables; separate must-have from nice-to-have so candidates self-assess accurately). Rules:

• Must-haves: 4 - 6 items maximum, each one the team would genuinely reject an otherwise-strong candidate for lacking. Use the competency dimensions from rubric.md (opens in new window) Step 2 for the role (e.g., for automation: test analysis and design, test code conventions, tooling depth, communication) so JD and rubric stay one vocabulary.
• Nice-to-haves: 3 - 5 items, explicitly labeled as such.
• Include the generic tester skills only when they will be screened for: CTFL 1.5.1 lists testing knowledge, thoroughness, curiosity, attention to detail, being methodical, and communication among skills particularly relevant for testers - but as JD boilerplate they screen nothing; tie them to an observable ("bug reports that developers can reproduce first try") or leave them out.
• Certifications (ISTQB included) are nice-to-have evidence of knowledge, not must-have proxies for skill, unless a client or regulator mandates them.

Step 4 - Define screening signals for the recruiter

The JD ships with a one-page screening note (internal, not posted): for each must-have, what in a CV or portfolio counts as signal vs noise. Example for "tooling depth, Playwright":

This note is what keeps the recruiter's screen consistent with the panel's rubric - the same chain-of-custody idea the structured-interview triple applies after the screen.

Step 5 - Assemble and length-check the JD

Order: title, one-paragraph role summary (which includes why the role is open), responsibilities, must-haves, nice-to-haves, team and stack, process and timeline, compensation per jurisdiction rules. Target 300 - 660 words total per Workable's guidance; bulleted lists for mobile readability. Worked example (condensed):

# Senior QA Automation Engineer - Payments

We run weekly releases for a payments product used by 40k merchants. This role
is open because our capability review found one engineer covering performance
testing for the whole group; you will broaden and own that coverage. ~80%
hands-on engineering, ~20% strategy input for your area.

## What you will do
- Design and automate regression tests for payment-retry and reconciliation flows
  (TypeScript + Playwright, k6 for load profiles).
- Extend the CI quality gates: flake quarantine, suite budget, pass-rate reporting.
- Run risk-based test analysis on new payment features with the product trio.
- Coach two mid-level engineers on test design through review.

## Must have
- Test analysis and design: you can derive tests from risks and requirements,
  not only from acceptance criteria handed to you.
- Production-grade test code in TypeScript or a near language; you have owned
  a suite others contribute to.
- Load or performance testing on at least one real system (k6, Gatling, or similar).
- Written communication: bug reports and strategy notes that stand alone.

## Nice to have
- Payments or other regulated-domain background; ISTQB CTAL-TA; CI ownership
  (GitHub Actions); accessibility testing exposure.

(~420 words in full form, inside the 300 - 660 band.)

Anti-patterns

Limitations

• No compensation benchmarking. Salary bands come from the org's compensation source; only enforce that disclosure rules per jurisdiction are respected in the posting.
• No legal review. Employment-law phrasing (at-will clauses, accommodation language) varies by jurisdiction and is HR/legal's call.
• Channel strategy out of scope. Where to post and how to source is recruiting strategy; this produces the artifact, not the campaign.
• The JD ages. Re-derive from current team context per opening; a reposted two-year-old JD advertises a team that no longer exists.

References

• ISTQB CTFL Syllabus v4.0, section 1.4.5 "Roles in Testing" (testing role vs test management role definitions quoted above) and 1.5.1 "Generic Skills Required for Testing": https://istqb.org/wp-content/uploads/2024/11/ISTQB_CTFL_Syllabus_v4.0.1.pdf - syllabus text verified 2026-06-10 from the published v4.0 PDF (ISTQB resource CDN copy at https://d288qud2qgn4l3.cloudfront.net/media/resources/ISTQB_CTFL_Syllabus-v4.0.pdf).
• Workable, "How to write a good job description" - title guidance, 300 - 660 word target, bulleted typical-workday duties, must-have vs nice-to-have split: https://resources.workable.com/tutorial/how-to-write-a-good-job-description (fetched 2026-06-10).
• interview-questions.md (opens in new window), rubric.md (opens in new window), calibration.md (opens in new window), onboarding.md (opens in new window) - the downstream hiring chain.

Onboarding plan authoring (30-60-90)

Deep reference for qa-hiring-kit SKILL.md - the post-hire ramp artifact. It starts at offer acceptance, not during the interview loop.

Why a structured ramp

Hiring ends at offer acceptance; onboarding is where the competency signal from the rubric is converted into a development plan. The 30-60-90 day framework - originally popularized by Michael Watkins' The First 90 Days (2003, Harvard Business Review Press) as the canonical structured-transition framework for new organizational members - divides the ramp into three equal phases: learn (days 1-30), integrate (days 31-60), and contribute independently (days 61-90). Each phase has a distinct focus, observable exit criteria, and a handoff to the next. (Notion blog on the 30-60-90 day plan framework: https://www.notion.com/blog/30-60-90-day-plan; Wikipedia, "Onboarding": https://en.wikipedia.org/wiki/Onboarding.)

The plan anchors each phase's competency targets to the six rubric axes from rubric.md (opens in new window): test analysis and design, defect lifecycle, test code conventions (automation roles), tooling depth, communication, and domain reasoning. Rather than treating onboarding as a generic HR checklist, the plan treats it as a continuation of the structured-interview signal - a hire scored 2 on "test code conventions" during the loop gets targeted development investment in exactly that axis during phase 2.

The PractiTest 2026 State of Testing Report found that nearly 40% of individual contributors feel "test strategy" is underdeveloped in their teams, and that practitioners who pivot toward strategy earn a +10.6% income premium versus those who remain in pure technical execution (https://www.practitest.com/state-of-testing/). The day-61-90 phase directly addresses this gap by pushing the hire toward strategy ownership proportional to seniority.

Step 1 - Capture the inputs

If rubric scores are not available (e.g., an informal hire), treat all six axes as equally weighted development targets and flag this assumption explicitly in the output.

Step 2 - Determine the seniority multiplier

Seniority changes the pace and scope of the plan, not its three-phase structure. The table below describes the expected exit state for each seniority at day 90 - the plan's success criteria derive from this:

[author opinion] These exit criteria follow the general "learn - integrate - contribute" arc described in the 30-60-90 day literature, adapted to QA-specific observable outputs. The pace should be recalibrated if the team's domain or compliance environment (e.g., healthcare, finance) has an unusually long environment-access ramp.

Step 3 - Author the three-phase plan

For each seniority, emit three sections. The structure below is for a mid qa-automation-engineer; adjust the target values per the multiplier table in Step 2 and the rubric scores from Step 1.

Phase 1 (days 1-30): Learn

Focus: environment, context, and first contribution. Per the 30-60-90 day framework (https://www.notion.com/blog/30-60-90-day-plan), this phase centres on acquiring the knowledge and role clarity needed to begin contributing - not on producing output.

Milestones:

1. Development environment running and verified against the team's setup doc by day 3.
2. All repository access, CI pipelines, and test-infrastructure credentials provisioned by day 5.
3. Read the team's test conventions document and the project's domain README by day 7.
4. Paired with mentor for a 1-hour walkthrough of the existing test suite by day 7.
5. First test case authored (existing coverage gap identified by the mentor) passing in CI by day 14.
6. Attended one sprint planning, one defect triage, and one retrospective by day 21.
7. Written a bug report for a defect found during exploratory testing, reviewed by a senior QA or engineer by day 28.
8. Completed ISTQB Foundation Level Chapter 1 self-study (Fundamentals of Testing) if not already certified (ISTQB CTFL v4.0 syllabus: https://astqb.org/certifications/foundation-level-certification/) by day 30.

Competency targets (rubric axes - first contact):

Mentor cadence (phase 1): 30-minute daily check-in for the first 2 weeks; 1-hour weekly thereafter. Mentor reviews all authored tests before merge.

Success criteria for phase-1 exit: First test merged to the main test suite; bug report accepted without revision; mentor confirms independent environment operation.

Phase 2 (days 31-60): Integrate

Focus: applying skills under supervision and owning a bounded scope. Per the 30-60-90 day framework, this phase shifts toward integration with the team and application of the new hire's existing skills, moving from supervised contribution to scoped ownership.

Milestones:

1. Takes ownership of one feature's test suite - additions, maintenance, and CI failures - by day 35.
2. Attends all sprint test-planning sessions and proposes at least one test case per story by day 40.
3. Reviews one teammate's test PR and provides written feedback by day 45.
4. Identifies and documents one test-architecture gap (e.g., a boundary class with no negative tests) by day 50.
5. Runs a 30-minute knowledge-share session for the QA or engineering team (format: what I learned in month 1) by day 55.
6. Completes ISTQB Foundation Level Chapters 2-4 self-study (Test Activities and Roles; Static Testing; Test Analysis and Design) if not certified by day 60.

Competency targets (rubric axes - applied):

Targeted development for rubric-axis score of 2 (borderline at hire): For each axis where the hire scored "borderline" during the loop, the plan adds a specific paired-learning task: e.g., a score-2 on "test code conventions" triggers a required code-review pairing with a senior on two consecutive PRs; a score-2 on "defect lifecycle" triggers a defect-retrospective exercise where the hire re-examines three historical bug reports and identifies the point of detection vs. the point of failure.

Mentor cadence (phase 2): 1-hour weekly; mentor shifts from prescriptive reviewer to code-review approver. Mentor stops blocking on approvals by day 45 (the hire uses the team's standard PR review process).

Success criteria for phase-2 exit: Owned feature suite green in CI with no unreviewed failures; architecture-gap document shared and acknowledged by the team; knowledge-share delivered.

Phase 3 (days 61-90): Contribute independently

Focus: independent contribution and the beginning of strategy ownership proportional to seniority. Per the 30-60-90 day framework, this phase prepares the hire to lead their first project or initiative - moving from "applying skills" to "driving decisions."

Milestones:

1. Leads test planning for one sprint end-to-end (story estimation, test-case authorship, risk identification) by day 70.
2. Produces a one-page test-coverage assessment for an assigned component, with a risk-ranked gap list, by day 75.
3. Proposes one process or tooling improvement with a concrete rationale and presents it at the team's retro by day 80.
4. Completes ISTQB Foundation Level Chapters 5-6 self-study (Managing the Test Activities; Test Tools) if not certified by day 85.
5. First 30-minute solo 1:1 with hiring manager at day 90: hire drives the agenda using the success criteria below.

Competency targets (rubric axes - owned):

Mentor cadence (phase 3): Bi-weekly 30-minute check-in; mentor role shifts from reviewer to consultant. New hire sets the agenda. Formal mentor relationship closes at day 90; standard peer-review process applies thereafter.

Success criteria for phase-3 (day-90) exit - the success criteria double as the hiring manager's 90-day review inputs:

1. Owns at least one test suite end-to-end with no unresolved CI failures.
2. Has led at least one sprint's test planning with documented coverage decisions.
3. Has produced at least one written artifact (coverage assessment, process proposal, or knowledge-share) that other team members reference.
4. Rubric-axis score improvements: any axis scored "borderline" at hire has a documented development event (pairing session, retro exercise, PR review) with a mentor note on progress.
5. Peer feedback (from one engineer and one QA reviewer, gathered by the hiring manager): the hire is "someone I can hand a feature to."

Step 4 - Emit the plan

The output is a single markdown document with:

1. Header: hire name (or placeholder), role, seniority, start date, hiring manager, mentor.
2. Rubric-score summary: the per-axis hire scores from the structured interview, copied from the rubric output - these anchor the targeted-development items in phase 2.
3. Three phase sections (Step 3 structure per seniority).
4. Milestone tracker table: a flat checklist of all milestones with day targets and owner (hire / mentor / hiring manager) for easy status tracking.
5. Hand-off block: share the plan with the new hire on or before day 1 (it is not confidential - the hire should know the success criteria); hiring manager reviews phase-exit criteria at day 30, 60, and 90; any axis still "borderline" at day 60 gets a second targeted-development pairing before day 75; at day 90, archive the plan alongside the rubric output in the team's hiring record - the question bank, rubric, and onboarding plan together form the hiring-to-ramp provenance trail.

Anti-patterns

Limitations

• The plan assumes rubric scores exist. Without per-axis scores, the targeted-development section of phase 2 defaults to generic guidance and loses its main differentiation from a generic onboarding checklist.
• Domain-specific ramp times vary. In regulated industries (healthcare, finance), environment access and compliance training can extend phase 1 by 2-4 weeks. Flag this assumption and adjust milestones accordingly.
• Mentor availability is a hard dependency. The phase-1 daily check-in and phase-2 PR review cadence require a named mentor. If no dedicated QA mentor is available, substitute a senior engineer and flag the capability gap.
• The ISTQB self-study milestones are optional enrichment. The CTFL Foundation Level syllabus (https://astqb.org/certifications/foundation-level-certification/) covers test fundamentals, static testing, test analysis and design, test activities, and test tools. For hires who are already certified, replace these milestones with domain-specific reading or toolchain deep-dives.
• No fairness review. The success criteria are not audited for differential treatment across protected classes - that is the team's HR review.

References

• Michael Watkins, The First 90 Days (Harvard Business Review Press, 2003 / 2013) - the foundational 90-day structured-transition framework. The learn / integrate / contribute phase arc follows Watkins' "orienting - learning - acting" progression.
• Notion, "30-60-90 day plan" - three-phase framework description (learn / integrate / lead): https://www.notion.com/blog/30-60-90-day-plan
• Wikipedia, "Onboarding" - organizational socialization definition and four adjustment dimensions (role clarity, self-efficacy, social acceptance, organizational culture knowledge): https://en.wikipedia.org/wiki/Onboarding
• ISTQB Certified Tester Foundation Level v4.0 syllabus (ASTQB mirror) - competency chapters cited for self-study milestones (Ch. 1: Fundamentals; Ch. 2: Test Activities and Roles; Ch. 3: Static Testing; Ch. 4: Test Analysis and Design; Ch. 5: Managing Test Activities; Ch. 6: Test Tools): https://astqb.org/certifications/foundation-level-certification/
• PractiTest 2026 State of Testing Report - test strategy underdevelopment (40% gap) and the strategy-vs-execution income premium (+10.6%) that motivates the phase-3 strategy-ownership milestones: https://www.practitest.com/state-of-testing/
• rubric.md (opens in new window) - the upstream reference whose six competency axes are the target axes for this plan's per-phase competency tables.
• calibration.md (opens in new window) - the calibration guide that closes the interview loop; the onboarding plan picks up where it ends (post-offer-acceptance).

Hiring rubric authoring

Deep reference for qa-hiring-kit SKILL.md - producing the competency-anchored scoring rubric, the how to score leg of the structured-interview tripod.

Why anchored rubrics

Without a rubric, two interviewers asking the same question produce different scores; the literature on structured interviewing (opens in new window) is clear that the questions alone are not sufficient - the scoring rubric is what converts them into a comparable signal.

Anchored rubrics outperform free-form scoring because the anchor descriptions at each level (no-hire / borderline / hire / strong-hire) constrain what each score means. An interviewer who reads "level 3: candidate explains the AAA pattern with a worked example and identifies one of: assertion strength, mocking pitfalls, or fixture coupling" cannot drift the score on tone or rapport - the anchor is concrete.

Step 1 - Capture the inputs

If a question bank is not available (e.g., an ad-hoc loop, or an existing interview set that was never written down), author one anchor set per competency dimension rather than per (competency × question) cell, mark the rubric provisional in the header, and flag this assumption explicitly. A provisional rubric must be re-run against the bank once it exists - competency-general anchors drift from the questions actually asked, which is the failure the requirement exists to prevent.

Step 2 - Pick the competency dimensions

A QA hiring rubric scores against 5 - 8 competency dimensions. The default set (drawn from ISTQB Foundation Level v4.0 (opens in new window) competencies and adapted to interviewable behaviour) per role:

manual-qa-engineer / qa-automation-engineer

1. Test analysis & design - partitioning, boundary, decision-table reasoning per ISTQB technique.
2. Defect lifecycle - defect vs failure distinction; bug-report quality; reproducibility.
3. Test code conventions (automation only) - AAA structure, assertion strength, mocking discipline.
4. Tooling depth - fluency with the team's primary toolchain (Playwright / Cypress / Selenium / pytest / JUnit / etc.).
5. Communication - written bug reports; verbal hand-off to engineering.
6. Domain reasoning - applies QA techniques to the team's domain (fintech / healthcare / consumer mobile).

sdet

1. Test analysis & design.
2. Test code conventions.
3. Test framework / tool architecture - how to extend the team's framework; CI integration; flake budget.
4. Production-quality coding - AAA, refactoring, naming, fixture cleanliness.
5. System reasoning - service boundaries; what to test at which layer.
6. Communication & collaboration.

test-lead

1. Test strategy authoring - risk-based testing; the test pyramid as an argument, not a template.
2. Stakeholder management - engineering, product, support, leadership.
3. Hiring & coaching of QA team members.
4. Defect management at the team / cross-team layer.
5. Tooling & CI ownership.
6. Communication (written + verbal, exec-level).

quality-manager

1. Quality strategy across releases / quarters.
2. Risk-based prioritisation - data-informed decisions with traceability.
3. Stakeholder communication, exec-level.
4. Hiring & team development.
5. Process / methodology fluency - agile, BDD, shift-left, shift-right, when each applies.
6. Defect / escape management at the org layer.
7. Compliance / regulated-industry framing (if applicable).

Emit the dimensions selected for the role; the team can add or remove dimensions before locking the rubric.

Step 3 - Author the 4-level anchors per dimension

For each (competency × question) cell, the rubric needs four behavioural anchors. The anchor describes what the candidate said or did, not what the interviewer felt - this is the load-bearing principle that reduces interviewer noise.

### Test analysis & design - Q3 (Behavioral, STAR: late-defect catch)

| Score | Anchor (what the candidate said / did) |
|---|---|
| **1 - no hire** | Cannot articulate a partition / boundary / decision-table technique. Describes the catch as "I just got lucky." Or attributes the catch to a tool ("the linter caught it"). |
| **2 - borderline** | Names one ISTQB technique correctly but cannot apply it to the catch they describe. STAR is partial: missing Result or missing the candidate's specific Action (says "we" throughout). |
| **3 - hire** | Identifies the specific technique that caught the defect (e.g., "we had no negative test for the empty-cart case - equivalence partitioning would have flagged it"). STAR complete: situation, task, the candidate's specific action, measurable result + retro learning. |
| **4 - strong hire** | Generalises beyond the specific defect: identifies a systemic gap (e.g., "we had no convention requiring a negative test per public method; I added that to our conventions doc"), and ties the change to a measurable downstream improvement. |

**Probe-trigger:** If the candidate scores 2 on STAR completeness, probe for the missing component; do not deduct further on the second pass.
**Time-budget impact:** A score of 4 typically takes 2 extra minutes; budget accordingly.

Each anchor is concrete enough that two interviewers reading the same transcript would arrive at the same score - that is the only test of the anchor's quality.

Step 4 - Compute the role-level summary score

The rubric outputs a per-dimension score and a summary recommendation. The summary is not a simple average:

The summary refuses to average across competencies - a candidate weak in defect lifecycle and strong in tooling depth is not "average"; the role demands both. Per-dimension floors are the load-bearing constraint.

Step 5 - Emit the rubric

The output is a single markdown document with:

1. Header: role, seniority, source question bank reference, competency dimensions, summary-rule table.
2. Per-question scoring sections (one per question in the bank, scoring against each competency the question targets - typically 1 - 2 competencies per question).
3. Summary recommendation rules.
4. Hand-off block: pair with calibration.md (opens in new window) for gold-standard model answers and common pitfalls per question - without those, the anchors here are aspirational. Run a calibration interview before the first real candidate (per the structured-interview research, calibration is the dominant variable in inter-rater agreement). Lock the rubric at the start of the hiring round; mid-round changes invalidate prior candidates' scores. After the round, retro the rubric: which competencies discriminated; which were noise; which scored everyone at 3 (a sign the anchor is too generous).

Anti-patterns

Limitations

• The rubric is only as good as its anchors. Vague anchors produce inter-rater drift; concrete behavioural anchors take time to author and refine.
• Anchor-validation requires real candidate data. Until the rubric has been used through 5 - 10 interviews, its anchors are theoretical. Plan a calibration interview before the first real candidate.
• Rubrics drift over time. A rubric may anchor on tools that are no longer the team's default. Re-author per hiring round, or at least review.
• No fairness audit. The rubric is not checked for bias against protected classes - that is the team's HR / legal review.
• Weighting is uniform per dimension. Some teams want to weight tooling depth higher than communication; emit unweighted scores and leave weighting to the hiring manager. Custom weights can be applied post hoc to the per-dimension scores.

References

• ISTQB Certified Tester Foundation Level v4.0 syllabus - the competency model adapted into the default dimensions per role: https://www.istqb.org/certifications/certified-tester-foundation-level
• ISTQB glossary - defect / failure distinction (load-bearing for the defect lifecycle dimension): https://glossary.istqb.org/en_US/term/defect
• Structured interview research - Levashina et al. 2014 (Personnel Psychology (opens in new window)) on the validity uplift from structured rubrics + same questions / same order.
• STAR behavioral interviewing method - Situation / Task / Action / Result framework, used in the behavioural-question anchors: https://en.wikipedia.org/wiki/Situation,_task,_action,_result
• Bloom's taxonomy - K1 - K4 cognitive levels used to align the rubric's anchor depth with the question's intended difficulty: https://en.wikipedia.org/wiki/Bloom%27s_taxonomy
• PractiTest 2026 State of Testing Report - hiring rubric authoring named as a high-adoption, low-risk AI use case for QA managers: https://www.practitest.com/state-of-testing/
• interview-questions.md (opens in new window), calibration.md (opens in new window) - the sibling references that complete the structured-interview triple.