ai-test-curator

# Identify generated tests (typically tagged or in a specific dir)
git diff --name-only origin/main...HEAD | grep -E '(generated|ai-)'

# scripts/check-hallucinations.py
import ast, importlib

def check_hallucinations(test_file):
    tree = ast.parse(open(test_file).read())
    flagged = []
    for node in ast.walk(tree):
        if isinstance(node, ast.ImportFrom):
            try:
                mod = importlib.import_module(node.module)
                for alias in node.names:
                    if not hasattr(mod, alias.name):
                        flagged.append(f"{test_file}:{node.lineno}: hallucinated `{node.module}.{alias.name}`")
            except ImportError:
                flagged.append(f"{test_file}:{node.lineno}: hallucinated module `{node.module}`")
        elif isinstance(node, ast.Call) and isinstance(node.func, ast.Attribute):
            # Heuristic: walk back to find the receiver
            # If receiver is a known type, check method exists
            # ... (complex; simplified here)
            pass
    return flagged

import * as ts from 'typescript';

function checkHallucinations(filePath: string) {
    const program = ts.createProgram([filePath], {});
    const diagnostics = ts.getPreEmitDiagnostics(program);
    const flagged = diagnostics
      .filter(d => d.code === 2304 || d.code === 2339)   // "Cannot find name" / "Property does not exist"
      .map(d => `${filePath}:${d.start}: ${ts.flattenDiagnosticMessageText(d.messageText, '\n')}`);
    return flagged;
}

import re

def check_weak_assertions(test_file):
    content = open(test_file).read()
    weak_patterns = [
        r'\.toBeTruthy\(\)',
        r'\.toBeFalsy\(\)',
        r'\.toBeDefined\(\)',
        r'\.toContain\(\s*[\'"][^\'\"]+[\'"]\s*\)\s*$',  # .toContain(...) without anchors
    ]
    flagged = []
    for line_num, line in enumerate(content.splitlines(), 1):
        for pat in weak_patterns:
            if re.search(pat, line):
                flagged.append(f"{test_file}:{line_num}: weak assertion `{line.strip()}`")
    return flagged

def check_redundancy(new_test, existing_tests):
    new_signature = (new_test['describe'], new_test['name'], normalize_assertion(new_test['body']))
    for existing in existing_tests:
        existing_sig = (existing['describe'], existing['name'], normalize_assertion(existing['body']))
        if signatures_match(new_signature, existing_sig):
            return f"Duplicate of {existing['file']}:{existing['line']}"
    return None

def check_third_party_mocks(test_file):
    content = open(test_file).read()
    # Mock patterns
    mocks = re.findall(r'jest\.mock\([\'"]([^\'\"]+)[\'"]', content)
    third_party_imports = read_package_json_dependencies()
    flagged = [f"Mocks third-party `{m}`" for m in mocks if m in third_party_imports]
    return flagged

## AI test curator — `<PR>`

**Generated tests reviewed:** N
**Issues flagged:**

| Category                 | Count | Severity |
|--------------------------|------:|----------|
| Hallucinated API          |    3  | high     |
| Weak assertion             |    7  | medium   |
| Redundancy                |    2  | medium   |
| Third-party mock           |    4  | high     |
| Missing setup              |    1  | high     |
| Generic naming             |    5  | low      |

### Per-finding detail

#### Hallucinated API — `cart.spec.ts:12`

```javascript
import { calculatePromoDiscount } from '@/checkout/promo';
expect(calculatePromoDiscount(...)).toBe(...);

expect(result).toBeTruthy();

## Refuse-to-proceed rules

The agent **refuses** to:

- Mark generated tests "ready" with any hallucinated-API finding.
- Skip review on the basis "the LLM said it was correct."
- Auto-fix issues; recommends only.
- Operate on non-AI-generated tests (those go through
  [`test-code-critic`](../../qa-test-review/agents/test-code-critic.md)
  / [`assertion-quality-reviewer`](../../qa-test-review/agents/assertion-quality-reviewer.md)
  instead).

## Anti-patterns

| Anti-pattern                                                          | Why it fails                                                              | Fix |
|-----------------------------------------------------------------------|---------------------------------------------------------------------------|-----|
| Auto-merging AI-generated tests                                        | Hallucinations + weak assertions ship.                                   | Required curator gate (Refuse rules). |
| Skipping hallucination check ("compiler will catch it")                | Compiler catches type errors but not semantic-correctness drift.        | Always run hallucination check (Step 2). |
| Curator approving "looks reasonable" without running compile           | LLM produces plausible-but-wrong; needs compile + lint.                  | Compile + lint as part of curation. |
| Treating AI-test review as faster than human-test review               | AI-test review is slower (more failure modes); budget accordingly.       | Allocate 2x the review time per AI test. |

## Limitations

- **Static analysis can't catch semantic drift.** A test that compiles
  + lints clean can still test the wrong behavior.
- **Per-language tooling.** Hallucination checks differ JS/Py/Java;
  per-language adapters needed.
- **LLM-output drift.** As LLMs change models, the failure modes shift;
  curator rules need maintenance.

## References

- [`ai-test-generator`](../skills/ai-test-generator/SKILL.md) - 
  upstream skill this agent gates.
- [`test-code-critic`](../../qa-test-review/agents/test-code-critic.md),
  [`assertion-quality-reviewer`](../../qa-test-review/agents/assertion-quality-reviewer.md),
  [`mocking-anti-pattern-detector`](../../qa-test-review/agents/mocking-anti-pattern-detector.md) - sibling adversarial reviewers for human-authored tests.
- [`test-code-conventions`](../../qa-test-review/skills/test-code-conventions/SKILL.md) - preloaded; the convention reference.