data-contract-extractor

Overview

This skill formalizes the prose of a data PRD into a data contract - the producer/consumer agreement on schema, freshness, volume, semantic invariants, and ownership - that the QA test suite can mechanically assert against, typically via dbt-tests, great-expectations, or soda-checks (see the qa-data-quality plugin). "Data contract" is practitioner-emergent terminology (Andrew Jones / Chad Sanderson), not ISTQB-canonical; this skill cites industry-engineering sources.

When to use

• A new data product (dbt model, table, file feed) is being scoped and the team wants assertions before shipping.
• An existing data product has implicit assumptions consumers rely on; the team is documenting them ahead of a refactor.
• A PRD or design doc references a dataset and the team needs the schema / SLA pulled out into structured form.
• A data-quality engineer needs the contract as input to suite generation.

What goes in a data contract

The five required sections, in order of authoring priority:

1. Schema

For each column:

2. Freshness SLA

3. Volume bounds

4. Distribution invariants

For each business-meaningful column, what must hold:

5. Ownership and governance

Output format

Emit as YAML for direct consumption by dbt / GX / Soda:

# data-contracts/<dataset-slug>.yml
contract_version: 1
dataset:
  name: orders
  description: One row per customer order; updated within 1h of placement.

schema:
  - name: order_id
    type: BIGINT
    nullable: false
    pk: true
    unique: true
    pii: none
    description: Surrogate key.
  - name: customer_id
    type: BIGINT
    nullable: false
    fk: customers.customer_id
    pii: indirect
    description: Customer who placed the order.
  - name: email
    type: VARCHAR
    nullable: false
    pii: direct
    constraint: { format: email }
    description: Customer email at time of order (immutable snapshot).
  - name: status
    type: VARCHAR
    nullable: false
    constraint:
      accepted_values: [placed, shipped, completed, returned]
    description: Current fulfillment status.
  - name: discount_pct
    type: DECIMAL(5,2)
    nullable: true
    constraint: { range: [0, 100] }
    description: Promotion discount applied; null = no promo.
  - name: updated_at
    type: TIMESTAMP
    nullable: false
    description: Last-modified timestamp; freshness source-of-truth.

freshness:
  cadence: hourly
  max_staleness: 2h
  source_column: updated_at

volume:
  min_per_day: 100
  max_per_day: 1000000
  volatility: cyclical
  recovery_action: |
    Investigate ingestion pipeline at <runbook-url> if volume falls outside bounds for >2 cycles.

distribution:
  - column: status
    rule: accepted_values
    values: [placed, shipped, completed, returned]
  - column: discount_pct
    rule: range
    range: [0, 100]
  - column: cancellation_rate
    rule: rolling_window
    window: 7d
    max: 0.05

ownership:
  producer_team: data-platform
  owner: '@data-platform-oncall'
  consumers:
    - dbt: marts/orders_summary
    - dashboard: 'Order Health'
    - service: shipping-microservice
  versioning: |
    Breaking changes coordinated via #data-contracts Slack 14d in advance.

Step-by-step extraction

When reading a data PRD:

1. Schema - look for table-spec sections, ER diagrams, or per-column descriptions. If only column names are given, flag missing types as gaps; do not guess.
2. PII tagging - every column gets a pii: tag, even pii: none. Force the data-product author to confirm; PII handling drives downstream architecture.
3. Freshness - look for "updated daily" / "real-time" / "<X minutes." If absent, flag as a gap.
4. Volume - look for traffic projections or current scale. If absent, flag as a gap to be filled before the contract is actionable.
5. Distribution - look for business rules in prose form ("status is one of...", "discount up to 100%") and translate.
6. Ownership - look for the responsible team / Slack channel.

Gap flagging

The agent never fabricates contract fields. Every gap becomes an explicit question:

## Contract gaps (HUMAN INPUT REQUIRED)

| Section       | Field                  | Question                                                                 |
|---------------|------------------------|--------------------------------------------------------------------------|
| schema        | `customer_phone.pii`   | Direct PII (yes - phone numbers are PII per most jurisdictions). Confirm. |
| freshness     | `cadence`              | PRD says "fresh data" - daily? hourly? real-time? Each implies a different gate. |
| volume        | `max_per_day`          | Not specified; required to set a `row_count between` assertion.          |
| distribution  | `payment_method`       | What's the accepted-value set? PRD lists four; are there more (e.g. "apple_pay" added recently)? |

Until the gaps are filled, the contract is incomplete and the test suite cannot be generated.

Examples

Three worked examples (PRD to contract, minimal source with a gap list, and refactor of an existing dataset) are in references/examples.md.

Anti-patterns

References

• ISTQB Glossary V4.7.1 - testability + non-functional testing (cited in non-functional-requirement-extractor) - for the underlying observability heuristic.
• ISO/IEC 25012:2008 - data-quality model (cite by stable ID; paywalled at iso.org).
• The qa-data-quality plugin - the downstream consumer of this skill's output. A data-quality engineer reads contracts produced by this skill.
• data-quality-conventions (in the qa-data-quality plugin) - naming and threshold conventions referenced from the contract.