attack-surface-test-checklist
Maps a code change to the security tests worth running against it. Classifies changed paths and file contents into nine attack surfaces (authentication, session management, input handling, file upload, deserialization, access control, API and web service, cryptography, data protection), attaches the matching OWASP ASVS 4.0.3 verification requirements, OWASP Top 10 2021 category IDs, and OWASP WSTG section numbers to each active surface, then emits a per-surface manual and automated test checklist bounded by what actually changed. Surfaces with no changed lines are excluded rather than carried as filler. Use when a pull request, release branch, or feature is about to be security tested and the team needs a targeted test list instead of a generic application-wide checklist.
Install with skills.sh (any agent)
npx skills add testland/qa --skill attack-surface-test-checklistattack-surface-test-checklist
Overview
A change set has an attack surface: the set of security-relevant behaviors its files participate in. A migration adding an avatar_url column and one adding a password_reset_token column look identical in a diff stat and need entirely different tests. This skill turns a change set into a bounded security test checklist in four moves:
Differentiation axis
What this owns: the mapping from a specific change to the specific security tests worth running against it, expressed as checkable items with standards references.
What it deliberately is not:
Which ASVS version this uses
All requirement IDs below use OWASP ASVS 4.0.3 numbering. ASVS renumbers requirements between major versions, so an ID with no version attached is ambiguous. ASVS 5.0.0 was released on 2025-05-30 and 4.0.3 remains published as the previous stable release (OWASP ASVS project page (opens in new window)). On a project standardized on 5.0.0, treat the chapter names below as the lookup key and re-resolve the numbers against that edition; the surfaces and test items do not change.
ASVS defines three verification levels: Level 1 is a low assurance level that is completely penetration testable, Level 2 is for applications containing sensitive data and is the recommended level for most applications, and Level 3 is for the most critical applications such as those handling high value transactions or sensitive medical data (ASVS 4.0.3, Using the ASVS (opens in new window)). For a per-change checklist, default to Level 1 items and escalate to Level 2 when the change touches credential storage or session token issuance. That default is a scoping convention of this skill, not an ASVS rule.
The nine attack surfaces
Classify each changed file by path first, then confirm or correct with content. A file can belong to more than one surface.
| Surface | Path signals | Content signals |
|---|---|---|
| Authentication | auth/, login/, oauth/, sso/, mfa/, token/ | Password hashing, session creation, JWT issuance, credential validation |
| Session management | session/, cookie, middleware/ | Cookie attribute setting, token expiry, invalidation on logout |
| Input handling | routes/, controllers/, validators/, forms/, parsers/ | SQL queries, template rendering, shell invocation, XML or JSON parsing |
| File upload | upload/, storage/, media/, attachments/ | Multipart handling, MIME validation, storage path construction |
| Deserialization | serializ, marshal, pickle, yaml.load, JSON.parse | Object hydration from an untrusted source |
| Access control | permissions/, policy/, roles/, authz/, acl/ | Role checks, ownership assertions, resource-level guards |
| API and web service | api/, graphql/, soap/, rest/ | Schema validation, HTTP method checks, rate limiting |
| Cryptography | crypto/, cipher, hmac, hash, tls/ | Key generation, algorithm selection, IV or nonce handling |
| Data protection | pii/, gdpr/, models/, db/, cache/ | Sensitive fields stored in plaintext, logging of secrets, caching policy |
Two rules keep the classification honest:
Step 1 - Mark the active surfaces
For each changed file, record the surface or surfaces it matched and the count of changed lines attributed to each. A surface is active when at least one changed line lands in it. Keep the file list per surface: it is what makes the test items concrete later ("SQL injection probes on the two new query parameters in orders_controller", not "test for SQL injection"). Record the excluded surfaces explicitly too. A reader needs to see that file upload was considered and found untouched, otherwise they cannot tell whether it was skipped or forgotten.
Step 2 - Attach ASVS requirements
| Surface | ASVS 4.0.3 chapter | Requirement areas to pull items from |
|---|---|---|
| Authentication | V2 Authentication | V2.2 General Authenticator Security, V2.4 Credential Storage, V2.5 Credential Recovery (V2 (opens in new window)) |
| Session management | V3 Session Management | V3.2 Session Binding, V3.3 Session Termination, V3.4 Cookie-based Session Management (V3 (opens in new window)) |
| Input handling | V5 Validation, Sanitization and Encoding | V5.1 Input Validation, V5.2 Sanitization and Sandboxing, V5.3 Output Encoding and Injection Prevention (V5 (opens in new window)) |
| File upload | V12 Files and Resources | V12.1 File Upload, V12.2 File Integrity, V12.3 File Execution, V12.4 File Storage, V12.6 SSRF Protection (V12 (opens in new window)) |
| Deserialization | V5 Validation, Sanitization and Encoding | V5.5 Deserialization Prevention (V5 (opens in new window)) |
| Access control | V4 Access Control | V4.1 General Access Control Design, V4.2 Operation Level Access Control, V4.3 Other Access Control Considerations (V4 (opens in new window)) |
| API and web service | V13 API and Web Service | V13.1 Generic Web Service Security, V13.2 RESTful Web Service, V13.4 GraphQL (V13 (opens in new window)) |
| Cryptography | V6 Stored Cryptography | V6.2 Algorithms, V6.3 Random Values, V6.4 Secret Management (V6 (opens in new window)) |
| Data protection | V8 Data Protection, plus V7 Error Handling and Logging | V8.1 General Data Protection, V8.2 Client-side Data Protection, V8.3 Sensitive Private Data (V8 (opens in new window)); V7.1 Log Content (V7 (opens in new window)) |
Data protection spans two chapters on purpose. Secrets leaking into logs is a logging requirement (V7.1), not a data protection one, and a checklist that only reads V8 misses it.
Step 3 - Tag Top 10 2021 categories and WSTG sections
Three tagging notes that catch people out:
Step 4 - Per-surface test items
Pull only the sections whose surface is active. Rewrite each item with the specific files, parameters, and endpoints recorded in Step 1.
Authentication
Manual:
Automated:
The remaining eight surfaces (session management, input handling, file upload, deserialization, access control, API and web service, cryptography, data protection) follow the same Manual / Automated shape with their own ASVS 4.0.3, Top 10 2021, and WSTG citations. Pull the full catalog from references/per-surface-test-items.md.
Output format
Emit one Markdown block containing, in order:
Keep item 7 even when it feels redundant. It is what stops a green checklist from being read as a sign-off.
Worked example
Change set: a pull request on a document management service adding shared-link downloads. Changed files:
src/api/routes/share_links.py +142
src/services/share_link_service.py +96
src/storage/attachment_fetcher.py +54
src/models/share_link.py +31
migrations/0042_share_link.sql +18
tests/api/test_share_links.py +210Classification: api/routes/ and the new query parameters make input handling and API and web service active. storage/ plus a fetch by URL makes file upload active (its SSRF sub-area specifically). The model and migration add a token column, making data protection active. share_link tokens are generated in the service, making cryptography active. The link grants object access without a session, making access control active. No login, cookie, or deserialization code changed. Test files are excluded from classification.
## Security test plan - docs-service PR #814 - 3f9a1c2
**Surfaces touched:** input handling (142), API and web service (142),
access control (96), file upload / SSRF (54), data protection (49),
cryptography (96)
**ASVS target level:** L2 (the change issues a bearer-style access token)
**Produced:** 2026-07-19
### Input handling
- [ ] [MANUAL] SQL metacharacters on `?token=` and `?expires=` in `share_links.py`; confirm parameterized queries (ASVS 4.0.3 5.3.4; WSTG 4.7.5)
- [ ] [MANUAL] Positive allow-list validation on both new parameters (ASVS 4.0.3 5.1.3)
- [ ] [AUTO] Injection static-analysis rules scoped to `src/api/routes/`
### API and web service
- [ ] [MANUAL] Verb tampering on `/share/{id}`: PUT, DELETE, PATCH (ASVS 4.0.3 13.2.1; WSTG 4.7.3)
- [ ] [MANUAL] Wrong and missing `Content-Type` rejected (ASVS 4.0.3 13.2.5, 13.1.5)
- [ ] [MANUAL] `token` is not exposed in the URL path or logs (ASVS 4.0.3 13.1.3)
### Access control
- [ ] [MANUAL] Fetch another tenant's document id with a valid token (ASVS 4.0.3 4.2.1; WSTG 4.5.4)
- [ ] [MANUAL] Force an exception in the token guard; confirm deny (ASVS 4.0.3 4.1.5)
### File upload / SSRF
- [ ] [MANUAL] `attachment_fetcher.py` remote fetch against 169.254.169.254 and file:// ; confirm allow list (ASVS 4.0.3 12.6.1; A10:2021)
### Cryptography
- [ ] [MANUAL] Token generated via CSPRNG, not `random` (ASVS 4.0.3 6.3.1)
- [ ] [AUTO] Flag `Math.random` / `random.random` in `share_link_service.py`
### Data protection
- [ ] [MANUAL] Token column encrypted or stored hashed (ASVS 4.0.3 8.3.7)
- [ ] [MANUAL] No token value in the new log lines (ASVS 4.0.3 7.1.1)
### Surfaces excluded (no changed lines)
Authentication, session management, deserialization
### Findings already visible in the change
- `share_link_service.py:41` builds the token with `random.randint`. This is a defect now, not a test item. Route it to defect triage.
### Not established by this checklist
Completing these items is evidence that the listed tests ran against this change. It is not a statement that the change is secure, and it is not an ASVS L2 attestation.Anti-patterns
| Anti-pattern | Why it fails | Correct behavior |
|---|---|---|
| Applying the generic Top 10 list unchanged | Every change gets the same forty items and the team stops reading them | Filter to active surfaces in Step 1 and list the excluded ones explicitly |
| Expanding to a full application assessment | The checklist balloons past what anyone will execute, so nothing gets tested | Bound the plan by the changed surface. A file upload change does not trigger a full authentication review |
| Building the checklist from scanner output | That is triage of findings that already exist, a different input and a different decision | Build this before scanners run; feed findings to defect triage instead |
| Inventing ASVS requirement numbers | Fabricated IDs misroute the tester and destroy trust in every other ID on the page | Cite only IDs read from the published chapter; when unsure, name the chapter and section instead |
| Quoting requirement IDs without a version | ASVS renumbers across major versions, so a bare V2.4 is ambiguous | Write the version next to the ID, as in ASVS 4.0.3 2.4.3 |
| Marking an item PASS with no evidence | Produces false confidence that survives into release | Items are binary: tested with recorded evidence, or not tested |
| Burying an obvious defect as a checklist item | A hard-coded key becomes a task someone might do next sprint | Split it out as a finding in its own section and route it now |
| Treating a complete checklist as a security sign-off | The checklist covers the changed surface only, at one ASVS level | Keep the "not established by this checklist" line in the output |
Limitations
Source index
Every requirement ID, section number, and quoted phrase above carries its source link at the point it is used. These are the three roots those links come from, for anyone who wants to browse rather than verify one claim:
Per-surface test items
View source (opens in new window)Per-surface test items
Deep reference for the attack-surface-test-checklist SKILL.md, Step 4. The Authentication surface is kept inline in SKILL.md as the worked example of the item shape; this file carries the other eight surfaces. Pull only the sections whose surface is active, and rewrite each item with the specific files, parameters, and endpoints recorded in Step 1.
Session management
Manual:
Automated:
Input handling
Manual:
Automated:
File upload
Manual:
Automated:
Deserialization
Manual:
Automated:
Access control
Manual:
Automated:
API and web service
Manual:
Automated:
Cryptography
Manual:
Automated:
Data protection
Manual:
Automated:
Related skills
definition-of-done
The team's Definition of Done (DoD), both halves of the lifecycle: authoring and auditing. Explains the Scrum Guide's DoD definition ("a formal description of the state of the Increment when it meets the quality measures required for the product"), proposes a starter DoD with the 7-10 lines most teams need (code reviewed, unit tests, docs, AC met, deployed to staging, smoke passed, no a11y regressions, telemetry wired), emits a per-PR checklist a reviewer enforces, and audits work against an existing DoD line by line with repository evidence (review records, diffs, CI runs, coverage reports), tagging every line met, not met, or unverifiable - never passing a line on self-attestation. Use when the team doesn't have a DoD, wants to revise theirs, or is about to mark a story or PR done and nobody has checked the work against the committed checklist.
e2e-suite-budget
Caps E2E suite size by computing per-test ROI - (regressions caught × value) ÷ (runtime × flake rate × maintenance) - then ranks every end-to-end test and recommends which bottom-decile ones to retire, move to a lower layer, or fix. Use when CI is slow or E2E-dominated, flaky failures are rising, or quarterly to keep suite size within maintenance capacity. For strategic unit:service:UI layer ratios use test-pyramid-balancer, for the minimal per-deploy critical-path gate use smoke-suite-gate, and for quarantining flaky tests use flaky-test-quarantine; this prunes low-signal tests by ROI.
framework-choice-advisor
Reference catalog for picking a test automation framework or QA tool - covers Playwright / Cypress / Selenium / WebdriverIO / Appium / Espresso / XCUITest / RestAssured / Karate / k6 / Locust with side-by-side tradeoffs on speed, cross-browser, mobile, parallelisation, language support, ecosystem maturity, CI integration; a decision tree matching project NFRs to framework choice; and reference layouts for the chosen stack. references/ extends the same decision to commercial procurement (seven-axis vendor evaluation for TCM platforms, no-code tools, visual-regression services) and to recording the outcome (ADR-based tool-selection decision record with signal, one recommendation, flip conditions). This is the upstream selection step: it decides which tool to adopt, not how to configure one already chosen. Use when starting a new test-automation suite, evaluating commercial QA vendors, or writing down a tool decision.
post-mortem-author
Build-an-X workflow that produces a blameless post-mortem from an incident - captures the timeline (chronological event sequence with sources), root cause analysis (what + why, not who), impact (users / revenue / SLO debt), action items (with owners + due dates + measurable success criteria), and "what went well" (intentional). Per Google SRE: "Blameless postmortems are a tenet of SRE culture." Use after every user-visible incident, not just severe ones.
risk-matrix
The risk-based testing (RBT) umbrella: risk matrix and risk register authoring, likelihood x impact scoring, risk storming, calibration, and risk-to-test coverage mapping. Produces the per-feature / per-release matrix artifact (structured intake: feature, category, impact 1-5 by likelihood 1-5, score; heatmap; mitigations with owners and due dates), supporting lightweight and heavyweight (FMEA / Cost of Exposure) methods per RBT canon, plus a risk coverage mapping workflow that proves which tests, cases, or monitors back each registered risk. references/ carries the product-risk and project-risk register variants, the risk-storming facilitation guide, matrix calibration against observed defect data, and a register review checklist. Use for any risk-based-testing artifact: building a matrix or register, running a risk-storming session, calibrating ratings against defects, or mapping risks onto test coverage.
smoke-suite-gate
Build-an-X workflow for a critical-path smoke suite that runs in <5 minutes - picks the 5-15 highest-business-value journeys (login, hero flow, checkout, payment, primary read), implements as fast E2E or API tests, gates per-deploy, retries on transient failures with quarantine. Use as the canary-precursor or per-deploy verification gate; the team's "if this fails, the build can't proceed" floor.
test-case-from-live-feature
Build-an-X workflow that produces a test-case matrix from a **live, undocumented feature** - running app at a URL, screen recording, screenshot, or verbal brief - by combining structured exploration (Playwright trace / DevTools / accessibility tree) with the four canonical heuristic test-design models bundled in references/ (Bach's HTSM / SFDPOT product elements, Whittaker's How-to-Break-Software attacks, Bolton's FEW HICCUPPS consistency oracles, ISO/IEC 25010 quality characteristics). Output is a structured case matrix, not an exploratory session charter. Use when there is no story, no AC, and no documentation - only a live feature - or as the heuristic reference layer for zero-documentation test design.
test-case-ideation-from-story
Turns a thin or ambiguous story into a reviewable test list - a backlog item that is a short paragraph plus the click-through support recorded for themselves, a spec that is mostly a list of accepted formats, or a tech design pasted into the ticket while the last few releases still shipped missed cases. Takes the story or feature spec and emits a markdown test-case matrix, one row per case (id, title, precondition, steps, expected, tier), covering happy path, alternate paths, boundaries, and negative paths, before any test code is written. Output is the human-reviewable matrix that goes into TestRail / Qase / Xray, not Gherkin scenarios. Use when a story needs its cases enumerated and agreed before automation starts.
test-effort-estimation
Turns a list of testable areas plus a change-shape distribution into a PERT three-point test effort estimate, reporting every row as a range around the expected value rather than a single number, requiring a named assumptions ledger across six mandatory categories, and recommending a per-layer ownership split across developer, automation, and exploratory roles. Bundles the change-shape classifier (pure-logic / service-layer / ui-heavy / data-heavy from git-history path and content signals, with the relative per-layer cost model) as a reference, so the shape distribution the estimate consumes can be produced here too. Does not choose which tests to run or how deep coverage should go. Use when an epic or release has been broken into testable areas and someone is about to commit test capacity for a sprint, or when a change set needs its shape classified before planning.
test-pyramid-balancer
Build-an-X workflow that analyzes a repo's test mix (unit / integration / E2E counts + runtimes) and recommends rebalancing toward the test pyramid ratios per the change-set shape - pure-logic-heavy repo wants ~80/15/5; UI-heavy repo wants ~60/25/15. Detects 'ice-cream cone' (E2E-heavy) and 'hourglass' (integration-thin) anti-patterns. Use when the user asks about test distribution, test strategy, test balance, too many E2E tests, slow CI caused by tests, testing best practices, or rebalancing their test suite; also suitable for quarterly calibration of the test mix to codebase reality.
test-strategy-author
Authors a test strategy document (a master test plan) for a project, release, or feature - covers scope, in/out, test types per layer (unit / integration / contract / E2E / perf / security / a11y), risk-based test prioritization that maps top risks to test investment (per `risk-matrix`), tooling stack, environments, exit criteria, and ownership. Includes a risk-based test-planning workflow that turns a feature scope plus the risk matrix into a budgeted per-risk test plan with owners, effort estimates, and an explicit risks-not-addressed section. Use when a team needs the release-readiness artifact stakeholders sign off on before significant test investment, or a risk-prioritized test plan for a feature or quarter.