compliance-evidence-generator

Overview

Auditors need more than passing tests. They require structured artifacts that map each control to verified outcomes, carry timestamps, and establish chain of custody. Per the NIST Computer Security Resource Center glossary (csrc.nist.gov/glossary/term/chain_of_custody, sourced NIST SP 800-72), chain of custody is "a process that tracks the movement of evidence through its collection, safeguarding, and analysis lifecycle by documenting each person who handled the evidence, the date/time it was collected or transferred, and the purpose for the transfer."

Per ISACA's Interactive Glossary (isaca.org/resources/glossary), an artifact is "a form of objective evidence that is an output of the work being performed and the process being followed." Both definitions anchor this skill: the output is objective, output-of-work artifacts with documented custody - not raw logs and not mere coverage reports.

This skill differs from its sibling skills:

When to use

• An audit engagement opens (SOC 2 Type II, ISO 27001 surveillance, PCI DSS annual ROC, HIPAA OCR inquiry, GDPR supervisory authority request).
• A GRC platform requests evidence upload for controls that have no native integration.
• CI now produces test results but they are not yet mapped to control IDs.
• A previous audit finding cited "insufficient evidence" for a control that is actually tested.

Step 1 - Build the control-to-test map

Start by resolving which test (or test suite) owns each in-scope control. Controls come from the applicable framework:

Produce a YAML or JSON mapping file, not a spreadsheet. Spreadsheets cannot be diff-reviewed or version-controlled cleanly:

# control-map.yaml
controls:
  - id: CC6.1            # SOC 2 Trust Services Criterion
    framework: soc2
    description: "Logical and physical access controls"
    tests:
      - suite: access_control_tests
        test_id: test_mfa_enforced_for_all_admin_accounts
      - suite: access_control_tests
        test_id: test_offboarded_user_revoked_within_sla
  - id: "ISO-A.8.3"       # ISO 27001:2022 Annex A
    framework: iso27001
    description: "Information access restriction"
    tests:
      - suite: rbac_tests
        test_id: test_role_least_privilege_enforced
  - id: PCI-10.2.1        # PCI DSS v4.0.1 Requirement 10
    framework: pci_dss
    description: "Audit log entries generated per 10.2.1"
    tests:
      - suite: audit_log_tests
        test_id: test_all_required_event_types_logged

One control may reference multiple tests. One test may satisfy multiple controls. Both are valid; record them.

Step 2 - Run tests and capture structured results

Test output must be machine-readable. Ad-hoc terminal output is not evidence. Accepted formats: JUnit XML, pytest JSON report (via pytest-json-report), Jest JSON (via --json), or any format parseable by the evidence assembler in Step 4.

# pytest with JSON output (pytest-json-report)
pytest --json-report --json-report-file=results/test-run-$(date -u +%Y%m%dT%H%M%SZ).json

# Jest with JSON output
npx jest --json --outputFile=results/test-run-$(date -u +%Y%m%dT%H%M%SZ).json

# JUnit XML (e.g., from Maven / Gradle / Robot Framework)
# output path depends on build tool - pass to Step 4 parser

Timestamp the file name at collection time (UTC). The collection timestamp is the first chain-of-custody data point.

Step 3 - Collect supporting evidence artifacts

Automated test results are the primary evidence. Supporting artifacts provide auditors with context:

Screenshots (UI controls tests)

# In Playwright / Selenium tests, save on pass as well as failure:
def capture_evidence_screenshot(page, control_id, test_name):
    path = f"evidence/{control_id}/{test_name}_{datetime.utcnow().isoformat()}.png"
    page.screenshot(path=path)
    return path

Log excerpts (access / audit / system logs)

Excerpts must be bounded and labeled. Unbounded log dumps are not evidence; they are noise that increases auditor workload and reduces the signal value of the artifact:

def extract_log_excerpt(log_source, start_utc, end_utc, control_id):
    """Extract the log window relevant to a control test run."""
    return {
        "control_id": control_id,
        "log_source": log_source,
        "window_start": start_utc.isoformat(),
        "window_end": end_utc.isoformat(),
        "lines": fetch_log_lines(log_source, start_utc, end_utc),
        "collected_at": datetime.utcnow().isoformat(),
        "collector": "compliance-evidence-generator",
    }

CI pipeline exports

Export the CI run as a permanent artifact (GitHub Actions: actions/upload-artifact, GitLab: artifacts:paths, CircleCI: store_artifacts). The artifact URL or download reference goes into the chain-of-custody record (Step 5).

Step 4 - Assemble the control-evidence matrix

The matrix is the auditor's index. It maps every in-scope control to its evidence files with pass/fail status and collection metadata.

import json, datetime, pathlib

def build_evidence_matrix(control_map_path, test_results_path, artifact_dir):
    control_map = load_yaml(control_map_path)
    test_results = load_json(test_results_path)
    matrix = []

    for control in control_map["controls"]:
        row = {
            "control_id": control["id"],
            "framework": control["framework"],
            "description": control["description"],
            "status": "PASS",
            "tests": [],
            "artifacts": [],
        }
        for test_ref in control["tests"]:
            result = find_test_result(test_results, test_ref["suite"], test_ref["test_id"])
            row["tests"].append({
                "test_id": test_ref["test_id"],
                "outcome": result["outcome"],      # PASS / FAIL / SKIP / ERROR
                "duration_ms": result["duration"],
                "run_at": result["timestamp"],
            })
            if result["outcome"] != "PASS":
                row["status"] = "FAIL"
            # Attach any per-test artifact files
            for artifact in glob_artifacts(artifact_dir, test_ref["test_id"]):
                row["artifacts"].append(str(artifact))

        matrix.append(row)

    output = {
        "generated_at": datetime.datetime.utcnow().isoformat() + "Z",
        "generator": "compliance-evidence-generator",
        "controls": matrix,
    }
    pathlib.Path("evidence/control-evidence-matrix.json").write_text(
        json.dumps(output, indent=2)
    )
    return output

Render the matrix as both JSON (machine-readable for GRC platform upload) and Markdown (human-readable for auditor review):

def render_matrix_markdown(matrix):
    lines = [
        "# Control-Evidence Matrix",
        f"Generated: {matrix['generated_at']}",
        "",
        "| Control ID | Framework | Description | Status | Tests | Artifacts |",
        "|---|---|---|---|---|---|",
    ]
    for c in matrix["controls"]:
        tests = ", ".join(t["test_id"] for t in c["tests"])
        artifacts = ", ".join(c["artifacts"])
        lines.append(
            f"| {c['control_id']} | {c['framework']} | {c['description']} "
            f"| {c['status']} | {tests} | {artifacts} |"
        )
    pathlib.Path("evidence/control-evidence-matrix.md").write_text("\n".join(lines))

Step 5 - Write chain-of-custody notes

NIST SP 800-72 (the source of the chain-of-custody definition used above) requires documenting each person who handled the evidence, the date/time of collection or transfer, and the purpose of the transfer. Apply this to each evidence file:

{
  "evidence_file": "evidence/CC6.1/test_mfa_enforced_20260604T143000Z.json",
  "control_id": "CC6.1",
  "collected_at": "2026-06-04T14:30:00Z",
  "collected_by": "ci-runner:github-actions:run-12345",
  "collection_method": "automated-test-result-export",
  "ci_run_url": "https://github.com/org/repo/actions/runs/12345",
  "artifact_url": "https://github.com/org/repo/actions/runs/12345/artifacts/67890",
  "transferred_to": "GRC platform evidence upload",
  "transferred_at": "2026-06-04T15:00:00Z",
  "transferred_by": "alice@example.com",
  "purpose": "SOC 2 Type II audit evidence submission - observation period Q2 2026",
  "hash_sha256": "abc123...",
  "notes": "Test run triggered by merge to main; no manual intervention."
}

Hash each artifact file (SHA-256) and record the hash in the custody note. Any tampering after collection breaks the hash and invalidates the artifact. This is the automated-evidence equivalent of the forensic-evidence integrity requirement in NIST SP 800-72.

Step 6 - Bundle the evidence package

The evidence package is the deliverable the auditor or GRC platform receives. Standard layout:

evidence-package-<engagement>-<date>/
  README.txt                          # engagement context, collection period, contacts
  control-evidence-matrix.json        # Step 4 machine-readable matrix
  control-evidence-matrix.md          # Step 4 human-readable matrix
  chain-of-custody/
    custody-<control-id>.json         # Step 5 per-control custody notes
  artifacts/
    <control-id>/
      <test-name>_<timestamp>.json    # test result excerpt
      <test-name>_<timestamp>.png     # screenshot (if applicable)
      <test-name>_<timestamp>.log     # log excerpt (if applicable)
  raw/
    test-run-<timestamp>.json         # full test results export (Step 2)

Produce the bundle as a deterministic archive (tar + gzip or zip with stable sort order) so the archive hash is reproducible from the same inputs. Record the archive hash in the final custody note.

# Reproducible archive (GNU tar with --sort)
tar --sort=name -czf \
  "evidence-package-${ENGAGEMENT}-$(date -u +%Y%m%d).tar.gz" \
  evidence-package-*/
sha256sum "evidence-package-${ENGAGEMENT}-$(date -u +%Y%m%d).tar.gz" \
  > "evidence-package-${ENGAGEMENT}-$(date -u +%Y%m%d).tar.gz.sha256"

Step 7 - Upload to GRC platform

The assembled package feeds into whichever GRC platform the engagement uses. All three major platforms accept manual evidence upload when no native integration covers the control:

For controls with native integrations (e.g., Vanta's GitHub integration for CC8.1 change management), prefer the integration over manual upload. Use this skill only to fill gaps the integration cannot cover, or when the GRC platform is not yet in use.

Step 8 - CI integration

Evidence generation should run automatically on every merge to main (or nightly for continuous-monitoring controls):

# .github/workflows/compliance-evidence.yml
name: Compliance Evidence
on:
  push:
    branches: [main]
  schedule:
    - cron: "0 2 * * *"     # nightly UTC

jobs:
  evidence:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4
      - name: Run compliance test suite
        run: pytest compliance_tests/ --json-report --json-report-file=results/test-run.json
      - name: Build evidence package
        run: python scripts/build_evidence_package.py
      - name: Upload evidence artifact
        uses: actions/upload-artifact@v4
        with:
          name: compliance-evidence-${{ github.run_id }}
          path: evidence-package-*/
          retention-days: 365   # retain for full observation period + buffer

Set retention-days to cover the audit's observation period plus a buffer. PCI DSS v4.0.1 Requirement 10 requires log and evidence retention of at least 12 months with the most recent 3 months immediately available.

GDPR accountability note

GDPR Art. 5(2) (gdpr-info.eu/art-5-gdpr) places the burden of proof on the controller: it "shall be responsible for, and be able to demonstrate compliance with, paragraph 1." Art. 24 (gdpr-info.eu/art-24-gdpr) requires controllers to "implement appropriate technical and organisational measures to ensure and to be able to demonstrate that processing is performed in accordance with this Regulation." Art. 30 (gdpr-info.eu/art-30-gdpr) requires maintaining a written record of processing activities available to supervisory authorities on request. The evidence package is the primary mechanism for satisfying all three provisions when controls are tested by automated means.

Anti-patterns

Limitations

• This is a build-an-X workflow. The control-map file and assembler script must be authored per engagement; there is no universal control catalog that maps to all frameworks without customization.
• NIST IR 8011 Vol 1 (csrc.nist.gov/publications/detail/nistir/8011/vol-1/final) defines automated assessment using "testable defect checks" against SP 800-53 determination statements. This skill follows the same pattern (desired state vs. actual state from test results) but does not implement NIST IR 8011's full sub-capability taxonomy.
• GRC platform ingestion formats vary and change; verify the target platform's current upload requirements at the time of the engagement.
• Some controls (e.g., physical access, vendor risk) cannot be tested automatically; those require manual attestation and are out of scope here.
• AICPA Trust Services Criteria (2017 with 2022 revised points of focus) are behind a registration wall at aicpa-cima.com - cited by stable document title per authoring guidance; readers should access via AICPA account.

References

• csrc.nist.gov/glossary/term/chain_of_custody - NIST SP 800-72 chain-of-custody definition (fetched 2026-06-04)
• isaca.org/resources/glossary - ISACA "artifact" and "audit evidence" definitions (fetched 2026-06-04)
• csrc.nist.gov/publications/detail/nistir/8011/vol-1/final - NIST IR 8011 Vol 1: automated assessment of SP 800-53 controls via testable defect checks (fetched 2026-06-04)
• gdpr-info.eu/art-5-gdpr, gdpr-info.eu/art-24-gdpr, gdpr-info.eu/art-30-gdpr - GDPR accountability obligations (fetched 2026-06-04)
• AICPA Trust Services Criteria 2017 (rev. 2022 points of focus) - available at aicpa-cima.com (paywalled; registration required)
• PCI DSS v4.0.1 Requirement 10 - 12-month retention requirement; source: pcisecuritystandards.org
• soc2-evidence-collector - SOC 2-specific raw log collection
• audit-trail-test-author - authoring tamper-evident audit log tests
• compliance-readiness-reviewer - coverage gap analysis agent