test-effort-estimation

Overview

A test effort estimate is a probability distribution over hours. Reporting it as one number destroys the only information that makes it useful: how wide the distribution is, and what has to stay true for it to hold.

This skill produces three things per epic:

1. A per-area, per-layer effort range computed with the PERT three-point method.
2. A ledger of named assumptions the ranges depend on.
3. A recommended split of which role owns which layer of the work.

Differentiation axis

This skill owns hours and ownership. It does not own the taxonomy of change shapes, the path and content signals used to detect them, or the relative per-layer cost model. Those belong to code-change-shape-classifier, and this skill consumes its output: a distribution over pure-logic, service-layer, ui-heavy, and data-heavy, each already mapped to the test layer where its verification lands. Do not re-derive that mapping here. If the shape definitions get copied into the estimator, the two answers drift and stop agreeing.

It also stops short of two downstream decisions. It does not select which existing tests to run for a given change, and it does not set coverage depth (how many tests of what type, with what entry and exit criteria). Both consume this estimate; neither is produced by it.

Read the boundary as: shape in, hours and owners out.

When to use

• An epic is broken into stories and someone is about to promise a sprint's worth of test capacity.
• A release plan needs a defensible number rather than a gut feel, and the number will be challenged.
• Two people disagree about how long a test area will take and a structured three-point round would settle it.
• A previous estimate was blown and the team wants to know which assumption failed, which is only answerable if a ledger existed.

Step 1 - Decompose into testable areas and weight them for risk

Split the epic into discrete testable areas. Each area maps to one or more acceptance criteria and to a coherent chunk of behaviour a person can be assigned and tracked against. Record for each:

• Area name: a short label such as "checkout flow", "discount-code API".
• Story IDs: which stories contribute to it.
• Change shape and layer: taken from the change-shape distribution, not re-derived. Each shape already carries the layer where most of its failure-detection value sits.

Then assign a risk weight on a coarse 1 to 3 scale:

Risk weight shifts effort upward: a risk-3 service area gets more test work than a risk-1 service area of the same size. The 1 to 3 scale is a practitioner convention used here as an effort modifier, not a standard risk-scoring method. It is deliberately too coarse to serve as a formal risk assessment. If the programme needs one, run a structured risk-scoring method separately and feed its output in as the weight.

An area with no risk weight is not estimable. Force at least one risk-1 and one risk-3 assignment per epic, or the scale collapses to "medium for everything" and stops carrying signal.

Step 2 - Elicit three points per area and layer

For each (area, layer) pair, produce three values in hours:

These are the standard three-point inputs: a is "the best-case estimate", m is "the most likely estimate", b is "the worst-case estimate" (Wikipedia, Three-point estimation).

Gathering the three points with more than one estimator

When several people could estimate the area, run a Wideband Delphi round rather than averaging opinions in a meeting. Barry Boehm and John A. Farquhar originated the wideband variant of the Delphi method in the 1970s, and called it "wideband" because it involved "greater interaction and more communication among those participating" than the traditional Delphi method (Wikipedia, Wideband delphi); the method was popularized through Boehm's Software Engineering Economics (1981) (same source).

Boehm's original sequence, adapted to test effort, is (Wikipedia, Wideband delphi):

1. The coordinator gives each estimator the area description and a blank form.
2. The group meets and discusses estimation issues, including what is in scope.
3. Estimators fill out a, m, b anonymously.
4. The coordinator summarizes and distributes the spread.
5. The group meets again, focusing only on the widely varying rows.
6. Estimators fill out the forms anonymously again. Repeat steps 4 to 6 as needed.

The anonymity in steps 3 and 6 is the mechanism, not a formality: it is what stops the loudest or most senior estimate from anchoring the rest. Every disagreement that surfaces in step 5 is a candidate assumption for Step 4's ledger, because two people estimating the same area differently almost always means they are assuming different things about scope, environment, or data.

Step 3 - Compute the expected value and the spread

Combine the three points with the PERT formulas:

E  = (a + 4m + b) / 6
SD = (b - a) / 6

The expected value is (a + 4m + b) / 6 and the standard deviation is (b - a) / 6, based on the assumption that a PERT distribution governs the data (Wikipedia, Three-point estimation). The same expected activity completion time, (t1 + 4t2 + t3)/6, and the same standard deviation, (t3 - t1)/6, are the classical PERT definitions in operations research, where the three estimates are combined "through the use of the beta and normal probability distributions" to describe how completion time might vary (Brunel University, Network analysis: uncertain completion times).

The 1:4:1 weighting is not a tuning knob. The weights "of 1/6:4/6:1/6 is essentially fixed and cannot be altered (as the underlying theory depends on these weights)" (Brunel University, Network analysis: uncertain completion times). If a team wants to weight the pessimistic case more heavily, that belongs in a larger b, not in a rewritten formula.

Report every row as a range, E - SD to E + SD. Never collapse a row to a point. The three estimates are not the only three outcomes; they define a distribution in which "all times are possible (with an associated probability)" (Brunel University, Network analysis: uncertain completion times). A single number silently discards that.

Aggregating rows to an epic total

Expected values add directly: the epic total E is the sum of the row E values.

Spreads do not. Sum the variances (SD squared) and take the square root of the total, which is how PERT combines activity spreads along a path (Brunel University, Network analysis: uncertain completion times):

E_total  = sum(E_i)
SD_total = sqrt( sum(SD_i^2) )

That aggregation assumes the rows are independent. The cited PERT treatment makes the same independence assumption explicitly and warns that when it does not hold "the probability figures calculated may be inaccurate" (Brunel University, Network analysis: uncertain completion times). Test rows are frequently not independent: one missing staging environment blows out every row at once. When rows share a dependency, also report the fully-correlated bound, sum(SD_i), as a worst case, and name the shared dependency in the ledger.

Step 4 - The assumptions ledger (mandatory)

An estimate without a ledger is not an estimate. It is a guess with a decimal point. The ledger is what makes the number auditable: it is the only artifact that lets someone later ask "which assumption failed?" instead of "who was wrong?".

This is not a local house rule. Federal cost-estimating guidance makes documentation one of the four pillars of a reliable estimate, and defines the well-documented pillar as: "Cost estimates need to identify rationales, assumptions, original source data, and methodologies used for calculations" (IRS Internal Revenue Manual 1.33.9.3, Four Main Pillars of Reliable Cost Estimates). The same guidance requires estimators to "Document the rationale and historical data supporting ground rules and assumptions", and defines a credible estimate as one that discusses and documents "data analysis limitations, risks, uncertainty, or special circumstances surrounding source data and assumptions" (same source, sections 1.33.9.3.2 and 1.33.9.3.4).

Every row in the effort table cites at least one assumption ID. Six categories are mandatory: an epic-level ledger that is missing any of them is incomplete, and the missing category is usually where the estimate later breaks.

Two rules make the ledger load-bearing:

• An estimate is a distribution, not a commitment. State plainly, in the output, that the range describes uncertainty and is not a delivery promise. When someone converts E into a date, they have made a commitment the estimate does not support.
• A violated assumption invalidates the estimate. It does not get padded. If assumption 2 fails because staging slips to day 6, do not add hours. Change the input and recompute the affected rows. Padding hides which assumption broke and destroys the audit trail that the ledger existed to provide.

Step 5 - Recommend the per-layer ownership split

Each (area, layer) row gets an owner. The default split below is a practitioner convention for a conventional team structure, not a standard; it is grounded in where the work naturally sits, not in a cited allocation rule.

Three allocation rules:

1. Every risk-2 and risk-3 area gets an exploratory row, with its own three points, owned by a manual tester. Exploratory time that is not estimated does not happen.
2. data-heavy areas get a dedicated data-checking row in addition to their service row: schema and contract changes fail in ways a request-level test does not see.
3. Risk-3 areas touching authentication, access control, or payment get a separate security-review row. Do not fold that time into the service row, because it is usually done by a different person on a different schedule.

Ownership is a recommendation. If capacity figures were supplied, compare each role's summed E against its available sprint hours and flag every role whose estimate exceeds capacity, with a suggested redistribution. If the split implies a large shift in the balance between layers (for example, an epic that adds 40 percent more UI-layer test work), say so explicitly so someone can review the overall test mix before the work starts.

Worked example

Epic: Promo codes at checkout (stories 12 to 15). Change-shape distribution supplied by the classifier: 30 percent service-layer, 25 percent ui-heavy, 25 percent pure-logic, 20 percent data-heavy. Three-point values gathered in a two-round Wideband Delphi with three estimators.

Row arithmetic, using E = (a + 4m + b) / 6 and SD = (b - a) / 6 (Wikipedia, Three-point estimation):

Checkout flow (service):    E = (4 + 32 + 16)/6 = 8.7    SD = (16 - 4)/6 = 2.0
Checkout flow (UI/E2E):     E = (2 + 20 + 10)/6 = 5.3    SD = (10 - 2)/6 = 1.3
Discount-code API (service):E = (2 + 16 +  8)/6 = 4.3    SD = ( 8 - 2)/6 = 1.0
Discount rules (unit):      E = (3 + 20 +  9)/6 = 5.3    SD = ( 9 - 3)/6 = 1.0
Promo schema (data checks): E = (2 + 24 + 14)/6 = 6.7    SD = (14 - 2)/6 = 2.0
Checkout (exploratory):     E = (2 + 12 +  6)/6 = 3.3    SD = ( 6 - 2)/6 = 0.7

Aggregation:

E_total  = 8.7 + 5.3 + 4.3 + 5.3 + 6.7 + 3.3 = 33.6 h
SD_total = sqrt(2.0^2 + 1.3^2 + 1.0^2 + 1.0^2 + 2.0^2 + 0.7^2)
         = sqrt(4.00 + 1.69 + 1.00 + 1.00 + 4.00 + 0.49)
         = sqrt(12.18) = 3.5 h
Independent-rows range:   30.1 to 37.1 h
Fully-correlated bound:   sum(SD) = 8.0  ->  25.6 to 41.6 h

The two ranges are reported together because assumption A2 (staging availability) is shared by four of the six rows, so the independence assumption behind SD_total is known to be imperfect (Brunel University, Network analysis: uncertain completion times).

Output format

Emit one Markdown document with these sections in order.

## Test effort estimate - Promo codes at checkout - 2026-07-19

**Total expected effort:** 33.6 h
**Range (independent rows):** 30.1 to 37.1 h
**Range (shared-dependency worst case):** 25.6 to 41.6 h

This is a distribution, not a commitment. It is invalidated, not padded, if any
assumption below changes.

### Effort by area and layer

| # | Area | Layer | Risk | a | m | b | E | Range | Owner | Assumptions |
|---|------|-------|-----:|--:|--:|--:|--:|-------|-------|-------------|
| 1 | Checkout flow | Service | 3 | 4 | 8 | 16 | 8.7 | 6.7 - 10.7 | Automation | A1, A2, A4, A6 |
| 2 | Checkout flow | UI / E2E | 3 | 2 | 5 | 10 | 5.3 | 4.0 - 6.6 | Automation | A1, A2, A5 |
| 3 | Discount-code API | Service | 2 | 2 | 4 | 8 | 4.3 | 3.3 - 5.3 | Automation | A2, A3 |
| 4 | Discount rules | Unit | 2 | 3 | 5 | 9 | 5.3 | 4.3 - 6.3 | Developer | A1, A3 |
| 5 | Promo schema | Data checks | 3 | 2 | 6 | 14 | 6.7 | 4.7 - 8.7 | Developer + data | A2, A4, A6 |
| 6 | Checkout | Exploratory | 3 | 2 | 3 | 6 | 3.3 | 2.6 - 4.0 | Manual tester | A1, A6 |

### Assumptions ledger

| ID | Category | Statement | Rows affected |
|----|----------|-----------|---------------|
| A1 | Scope boundary | Stories 12 to 15 only; story 16 (dark mode) excluded | 1, 2, 4, 6 |
| A2 | Environment | Staging available from sprint day 2 | 1, 2, 3, 5 |
| A3 | Test data | Fixture generator covers all discount-code cases | 3, 4 |
| A4 | Dependency | Payment provider sandbox contract unchanged this sprint | 1, 5 |
| A5 | Skill | One automation engineer with browser-automation experience | 2 |
| A6 | Risk rating | Checkout and promo schema rated risk-3: real payments, irreversible migration | 1, 2, 5, 6 |

All six mandatory categories present.

### Ownership summary

| Role | Rows | Expected hours | Capacity | Flag |
|------|------|---------------:|---------:|------|
| Developer (unit + data checks) | 4, 5 | 12.0 | 16 | ok |
| Automation engineer (service + E2E) | 1, 2, 3 | 18.3 | 14 | OVER by 4.3 h |
| Manual tester (exploratory) | 6 | 3.3 | 8 | ok |

### Capacity flags

Automation engineer is over-allocated by 4.3 h. Options: move row 3 to the
developer who owns the discount-code endpoint, or descope row 2 to the single
happy path and cover the variants in row 6.

### Method

Three-point PERT per row: E = (a + 4m + b) / 6, SD = (b - a) / 6. Row ranges
are E - SD to E + SD. Epic spread aggregated as sqrt(sum of variances), with the
fully-correlated sum reported alongside because A2 is shared across four rows.
Three-point inputs from a two-round Wideband Delphi with three estimators.
Change-shape distribution consumed as an input, not derived here.

Keep the "distribution, not a commitment" line and the "Method" section even when they feel redundant. They are what stop a reader from turning E into a date.

Anti-patterns

Limitations

• Three-point inputs are only as good as the estimator's domain knowledge. When nobody has built the area before, even b is usually too low. Record "first-time implementation" as a skill-category assumption and widen b deliberately.
• No historical velocity is used. The estimate derives from area structure, risk weight, and change shape, not from past sprint actuals. A team with a points-to-hours baseline should apply that conversion after Step 3 and record the conversion factor in the ledger.
• The PERT distribution assumption may not fit. The formulas rest on the assumption "that a PERT distribution governs the data" (Wikipedia, Three-point estimation), and a triangular distribution is a documented alternative for some applications (same source). Test work with a long tail of rare blockers is poorly described by any of them.
• Row independence rarely holds exactly, which is why the correlated bound is reported alongside. When one environment or one person gates most rows, the correlated bound is the honest number.
• The 1 to 3 risk weight is a coarse effort modifier, not a risk assessment. It cannot substitute for a structured risk-scoring method.
• The ownership split assumes a conventional structure with developers, automation engineers, and manual testers as distinct roles. A solo tester or a fully automated team collapses all rows onto fewer owners, which changes the capacity flags but not the hours.
• Non-functional test effort is out of scope. Load, security, and accessibility test work is scoped by their own methods; only the security review row above is reserved, and it is reserved rather than estimated.
• Predicted change shapes are weaker than measured ones. For an epic with no code yet, the distribution comes from story text. Record that in the ledger and recompute once code exists.

References

• Wikipedia, Three-point estimation: definitions of a, m, b; E = (a + 4m + b) / 6; SD = (b - a) / 6; the PERT distribution assumption and the triangular-distribution alternative.
• Brunel University, Network analysis: uncertain completion times: the classical PERT expected time (t1 + 4t2 + t3)/6 and standard deviation (t3 - t1)/6; use of the beta and normal distributions; the fixed 1/6:4/6:1/6 weighting; all times possible with an associated probability; variances summed and square-rooted; the independence assumption and its inaccuracy warning.
• Wikipedia, Wideband delphi: Barry Boehm and John A. Farquhar originated the wideband variant in the 1970s; named for "greater interaction and more communication among those participating"; popularized through Software Engineering Economics (1981); the six-step anonymous-round sequence.
• IRS Internal Revenue Manual 1.33.9, Cost Estimating Guidelines: the four pillars of a reliable estimate; the well-documented pillar requiring rationales, assumptions, source data, and methodologies; documenting the rationale behind ground rules and assumptions; the credible pillar requiring documented limitations, risks, and uncertainty.
• Fowler, TestPyramid: tests running end-to-end through the UI are "brittle, expensive to write, and time consuming to run".
• Fowler and Vocke, The Practical Test Pyramid: unit tests "run very fast"; integration with filesystems and databases is "much slower"; end-to-end tests "require a lot of maintenance and run pretty slowly"; exploratory testing is "a manual testing approach that emphasises the tester's freedom and creativity to spot quality issues in a running system".