ai-test-generator
Generates tests from natural-language specs (acceptance criteria, user stories, requirements) using an LLM, with confidence scoring per test case (LLM self-assessment plus heuristics: assertion quality, naming, completeness), batching uncertain cases for human review, and integration with the team's existing test framework. Use when the user asks to generate unit tests from acceptance criteria, convert user stories to test cases, automate test creation from requirements, or augment a spec-driven test suite with AI-generated stubs that are then curated before merge.
Install with skills.sh (any agent)
npx skills add testland/qa --skill ai-test-generatorai-test-generator
Overview
This skill provides an augmentation framework for converting acceptance criteria (ACs) into test code: AI generates, the team curates. Generated tests are scored for confidence and reviewed adversarially before merge.
Step 1 - Define the input
# input/cart-promo.yaml
spec_source: "stories/LIN-1234.md"
acceptance_criteria:
- id: AC-1.1
description: "Valid promo 'WELCOME10' reduces subtotal by 10%"
inputs:
cart_total: 24.99
promo_code: "WELCOME10"
expected:
subtotal_after: 22.49
message: "Code applied"
- id: AC-1.2
description: "Expired promo shows error 'This code has expired'"
inputs:
cart_total: 24.99
promo_code: "EXPIRED50"
expected:
subtotal_after: 24.99
error: "This code has expired"Step 2 - Run the generator
Generate one test per AC. Core loop below; full script with helpers and the project-conventions injection is in references/generation-and-scoring.md.
# scripts/ai-gen.py
import openai
system = ("Generate one test per AC in {framework} using the project's test "
"conventions (inject docs/test-code-conventions.md). Specific "
"assertions only - no .toBeTruthy() / .toBeDefined(). If an AC can't "
"be satisfied with the given inputs, mark CONFIDENCE: low and say why.")
for ac in input_yaml['acceptance_criteria']:
response = openai.chat.completions.create(
model='gpt-4',
messages=[
{'role': 'system', 'content': system.format(framework='jest')},
{'role': 'user', 'content': f"{ac['id']}: {ac['description']} | "
f"inputs {ac['inputs']} -> {ac['expected']}"},
],
)
open(f"tests/generated/{ac['id']}.test.js", 'w').write(
response.choices[0].message.content)Step 3 - Validate before scoring
Before scoring, verify that generated test files parse and compile. Failing tests caught here should be flagged as CONFIDENCE: low automatically.
# For TypeScript projects
npx tsc --noEmit
# For Python projects
python -m py_compile path/to/generated_test.py
# For JavaScript projects (syntax check)
node --check path/to/generated_test.jsAny file that fails compilation is immediately downgraded: subtract 50 from its score before applying the heuristics in Step 4.
Step 4 - Confidence scoring
Per generated test, compute a confidence score. The rubric is the core of the skill; its parsing helpers (extract_imports, module_exists, extract_test_name) are in references/generation-and-scoring.md.
def score(test_code, ac):
score = 100
if 'CONFIDENCE: low' in test_code: # LLM's own confidence
score -= 40
weak = ['.toBeTruthy()', '.toBeDefined()', '.toBeFalsy()', '.toContain(']
score -= sum(20 for m in weak if m in test_code) # vague matchers
for imp in extract_imports(test_code): # hallucinated imports
if not module_exists(imp):
score -= 30
name = extract_test_name(test_code).lower() # generic naming
if any(g in name for g in ['works', 'should', 'test 1', 'placeholder']):
score -= 15
return max(0, score)| Score | Action |
|---|---|
| 80-100 | High-confidence - review can be quick. |
| 50-79 | Medium - careful review required. |
| <50 | Low - likely needs rewrite or rejection. |
Step 5 - Output structure
## AI-generated tests - `<spec>`
**Generated:** N tests
**High-confidence:** M (review: spot-check 2-3)
**Medium-confidence:** K (review each)
**Low-confidence:** L (likely rewrite)
### High-confidence (4)
(test code blocks with confidence scores)
### Medium-confidence (3)
(blocks with confidence scores + flagged issues)
### Low-confidence (2)
(blocks with confidence scores + recommend manual rewrite)
### Hand-off
Review each generated test for:
- Hallucinated APIs / functions / constants
- Weak assertions (assert on specific expected values, not just truthiness)
- Missing setup / teardown
- Redundancy with existing tests
After curation: merge.Step 6 - Iteration loop
Spec → Generate → Validate → Score → Review → (rewrite | merge | reject)
↓
Lessons fed back into promptThe team's prompt evolves: when the LLM keeps producing .toBeTruthy(), add an explicit prohibition. When it hallucinates an API, add an example of the real API.
Step 7 - Cost + rate management
LLM calls have cost and rate limits. Pattern:
Anti-patterns
Limitations
References
Generation and scoring - full scripts
View source (opens in new window)Generation and scoring - full scripts
The SKILL.md spine keeps a minimal runnable core of each script. This file holds the production versions with their helper functions.
Generator (Step 2, full version)
# scripts/ai-gen.py
import openai
import os
system_prompt = """
You generate tests in {framework} for the given AC spec.
Constraints:
- One test per AC.
- Use the project's test code conventions (see test-code-conventions reference).
- Specific assertions only - no .toBeTruthy() / .toBeDefined() style.
- Use {test_runner}'s standard primitives.
- If you can't satisfy an AC with the given inputs, mark with
CONFIDENCE: low and explain why.
"""
def format_ac_prompt(ac):
"""Format a single AC dict into a prompt string for the LLM."""
return (
f"AC ID: {ac['id']}\n"
f"Description: {ac['description']}\n"
f"Inputs: {ac['inputs']}\n"
f"Expected: {ac['expected']}"
)
def save_test(ac_id, test_code, output_dir='tests/generated'):
"""Write generated test code to tests/generated/<ac_id>.test.js (or .py)."""
os.makedirs(output_dir, exist_ok=True)
# Detect language from shebang or content; default to .js
ext = '.py' if test_code.lstrip().startswith('def ') or 'import pytest' in test_code else '.js'
path = os.path.join(output_dir, f"{ac_id.replace('-', '_').lower()}{ext}")
with open(path, 'w') as f:
f.write(test_code)
return path
for ac in input_yaml['acceptance_criteria']:
response = openai.chat.completions.create(
model='gpt-4',
messages=[
{'role': 'system', 'content': system_prompt.format(
framework='jest', # replace with team's framework
test_runner='jest', # replace with team's test runner
)},
{'role': 'user', 'content': format_ac_prompt(ac)},
],
)
save_test(ac['id'], response.choices[0].message.content)The test-code-conventions reference in the system prompt should be a project-specific file (e.g. docs/test-code-conventions.md) injected into the prompt at runtime.
Scoring helpers (Step 4, full version)
The spine keeps the score() rubric inline; these are the parsing helpers it calls.
import importlib.util
import ast
import re
def extract_imports(test_code: str) -> list[str]:
"""Return a list of top-level module names imported in the code."""
try:
tree = ast.parse(test_code)
except SyntaxError:
return []
imports = []
for node in ast.walk(tree):
if isinstance(node, ast.Import):
imports.extend(alias.name.split('.')[0] for alias in node.names)
elif isinstance(node, ast.ImportFrom) and node.module:
imports.append(node.module.split('.')[0])
return imports
def module_exists(module_name: str) -> bool:
"""Return True if the module can be found in the current environment."""
return importlib.util.find_spec(module_name) is not None
def extract_test_name(test_code: str) -> str:
"""Return the first test/it/def test_ name found in the code."""
match = re.search(r'(?:it|test|def test_)\s*\(?["\']?([^"\'(),]+)', test_code)
return match.group(1) if match else ''Related skills
ai-spec-coverage-mapper
Build-an-X workflow that uses an LLM to map existing tests to spec sections - given a spec doc + the test suite, the LLM identifies which tests cover which sections, surfaces uncovered sections (gap), and recommends specific tests to add. Output is a coverage matrix per spec ID. Scope is mapping tests that already exist and naming the gaps, not authoring tests for new acceptance criteria. Use when a spec doc and a test suite both exist but nobody can say which requirements are actually covered - before a release sign-off, an audit, or a decision about where to spend the next round of test effort.
input-domain-coverage-audit
Audits a test file's input-domain coverage per entry point across three axes: equivalence partitioning (clustering the literal values the tests actually pass, to infer which partitions are exercised), boundary value analysis (recorded n/a when the entry point declares no bound), and error/negative-path coverage (classifying every matcher as positive or negative and computing the negative-assertion ratio). Emits a PASS / SHALLOW / N/A verdict per axis per entry point, with the evidence that produced it. Owns whether the test data spans the input space, not whether an individual assertion is specific enough: matcher specificity belongs to `test-code-conventions`. Use when a test file's cases all look alike - every argument the same shape, every response a success, no thrown-error case - and the suite needs a defensible answer on whether it exercises more than one equivalence class before it is approved.
model-based-test-graph-author
Build-an-X workflow for model-based testing (MBT) per the canonical definition - authors a state-machine model of the SUT (states + transitions + guards + actions), validates the model is connected and complete, and feeds the model to a test generator (manual / AI / dedicated MBT tool) that produces test paths covering each transition. Per Wikipedia (en.wikipedia.org/wiki/Model-based_testing): MBT "leverages model-based design for designing and possibly executing tests." Use when a complex stateful flow (checkout, onboarding, multi-step wizard) needs systematic coverage that ad-hoc tests miss.