soc2-evidence-collector

Overview

SOC 2 (Service Organization Control 2) is the AICPA-defined audit framework for SaaS providers. Per AICPA Trust Services Criteria (TSC):

Type I = point-in-time design assessment. Type II = continuous-evidence assessment over an observation period (3 - 12 months).

Type II requires evidence collection across the period - auditors sample, but every control should have continuous evidence available.

This is a build-an-X workflow - the per-criterion evidence collection script, not a standalone tool. Pair with Vanta / Drata / Secureframe (commercial GRC platforms) for evidence storage + auditor-facing dashboards.

When to use

• Pre-audit: Type II observation period started; need continuous evidence.
• Pre-pre-audit: identifying which controls + evidence are testable vs require manual attestation.
• Post-finding remediation: a previous audit flagged an evidence gap.
• Adopting a GRC platform (Vanta/Drata/Secureframe) and need evidence-feed configuration.

Step 1 - Identify in-scope criteria

Most SaaS engagements include CC + Availability + Confidentiality. Privacy criteria add when GDPR/CCPA also in scope. Processing Integrity adds for fintech / data-processing SaaS.

Step 2 - Auto-collect evidence per criterion

Map each control to one or more automatable evidence sources:

Example collector script:

# evidence/cc6_1_logical_access.py
import okta_client, datetime, json

def collect_cc6_1_evidence(start_date, end_date):
    """Per CC6.1: collect user-access audit events for the period."""
    events = okta_client.get_audit_events(
        type='user.session.start',
        start_date=start_date,
        end_date=end_date,
    )
    evidence = {
        'control_id': 'CC6.1',
        'period_start': start_date.isoformat(),
        'period_end': end_date.isoformat(),
        'evidence_type': 'user_access_logs',
        'sample_size': len(events),
        'events': events[:100],   # auditor sample
        'collected_at': datetime.datetime.utcnow().isoformat(),
        'collector': 'soc2-evidence-collector v1.0',
    }
    with open(f'evidence/cc6_1_{start_date.date()}_{end_date.date()}.json', 'w') as f:
        json.dump(evidence, f, indent=2)

Step 3 - Per-control test patterns

Beyond raw evidence collection, write tests that verify the control operates correctly:

def test_cc6_3_offboarded_user_has_no_active_sessions():
    """CC6.3: deprovisioned users must lose all access immediately."""
    user = User.objects.get(email='alice@example.com')
    deprovision(user)

    # Verify across all systems:
    assert not okta_client.user_has_active_sessions(user)
    assert not aws_iam.user_exists(user.aws_username)
    assert not github_org.is_member(user)
    assert not slack.is_member(user)
    # Audit log records the deprovisioning event:
    assert AuditLog.objects.filter(
        actor='hr-system',
        action='deprovision',
        subject=user.email,
    ).exists()

These tests run in CI; their pass/fail history is itself evidence for the auditor.

Step 4 - GRC platform alignment

Default: Vanta - the broadest native-integration coverage (AWS / Okta / GitHub / GSuite / etc.) means the auto-collected evidence (Step 2) only needs to fill gaps the integrations don't cover. Use the alternatives when Vanta doesn't fit:

Across all three, evidence ingest format is platform-specific but the auto-collected JSON (Step 2) feeds the platform's manual-upload UI when no native integration exists for your tooling.

Step 5 - Observation period sampling

Type II auditors typically request:

• Population list (all instances of a control event during the period - e.g., all PRs merged, all access-grants)
• Sample (auditor selects 25 - 40 random instances)
• Per-sample evidence (the specific logs, tickets, approvals)

Your evidence collector should support both:

• Population queries (SELECT * FROM audit_log WHERE date BETWEEN ...)
• Per-instance evidence retrieval (full context for one event)

Step 6 - Continuous-monitoring controls

Some controls are continuous (e.g., CC7.1 threat detection) - the evidence is an alert-history feed, not point-in-time samples.

Pattern: daily collector cron job that:

1. Queries the source system for the previous day's events
2. Stores in append-only evidence storage (S3 with versioning, immutable)
3. Records collector-run metadata (when, what, how many records)

Continuity gaps in collector runs are themselves audit findings - make collector failures alert-worthy.

Step 7 - End-to-end recipe

For each in-scope criterion:

1. ✅ Map criterion to evidence source(s)
2. ✅ Implement automated collector (Step 2)
3. ✅ Write per-control test (Step 3)
4. ✅ Wire evidence into GRC platform (Step 4)
5. ✅ Verify continuous-collection has no gaps (Step 6)
6. ✅ Run a mock auditor query (request a sample; verify response is complete + timely)

Anti-patterns

Limitations

• This is a build-an-X workflow. Tests use the team's existing test framework + cloud APIs.
• SOC 2 has many criteria; this skill is a starting framework, not a complete control library.
• Trust Services Criteria evolve (current TSP 2017, revised 2022); pin version per audit engagement.
• GRC platforms are commercial; OSS alternatives (e.g., comply-soc2 / Drata-compatible scripts) exist but are less polished.
• This skill doesn't replace a SOC 2 readiness consultant.

References

• aicpa.org/topic/audit-assurance/audit-and-assurance-greaterthan-suitable-trust-services-criteria - AICPA Trust Services Criteria (paywalled; free abstract)
• vanta.com/solutions/soc-2 - Vanta SOC 2 product
• drata.com/grc-central/soc-2 - Drata SOC 2 reference
• secureframe.com/hub/soc-2 - Secureframe SOC 2 hub
• gdpr-test-patterns, hipaa-test-patterns, audit-trail-test-author - sister test-pattern catalogs
• compliance-readiness-reviewer - agent