Testland
Skills & AgentsBlogAbout
Browse all skills & agents

contract-compatibility-gate

Builds a unified deployment-readiness gate that aggregates verdicts from any combination of Pact `can-i-deploy`, oasdiff (OpenAPI), graphql-inspector (GraphQL), and `buf breaking` (Protobuf), applies severity-aware pass/fail thresholds, and emits a single go / no-go decision with per-finding rationale. Use when authoring a CI step that gates a deployment on cross-protocol contract compatibility.

contract-compatibility-gate

Overview

Modern services rarely speak just one protocol. A single deployment might depend on Pact verifications (consumer/provider HTTP), OpenAPI compatibility for an external REST surface, GraphQL schema regressions for a public graph, and Protobuf breaking-change checks for internal gRPC. Each tool has its own exit code, its own severity levels, and its own concept of "breaking":

ToolVerdict surface
Pact can-i-deployExit 0 = yes / 1 = no (can-i-deploy)
oasdiff breakingExit driven by --fail-on ERR/WARN/INFO (oasdiff-breaking)
graphql-inspector diffNon-zero on BREAKING; zero on DANGEROUS / NON_BREAKING (gqi-diff)
buf breakingNon-zero on any rule violation in active categories (buf-breaking)

This skill defines a unified gate: collect each tool's structured output, normalize to one record shape, apply a single severity rule, emit one verdict.

When to use

  • A deployment depends on more than one protocol (typical for a service exposing both gRPC + REST, or a frontend consuming both GraphQL and REST).
  • The team wants the verdict in $GITHUB_STEP_SUMMARY (or equivalent) rather than four separate "❌ pact" / "❌ oasdiff" CI badges.
  • Some checks should be advisory rather than blocking - e.g. block on oasdiff ERR but let WARN pass with a comment.
  • The team wants per-tool ratchet behavior (existing failures grandfathered, new failures block).

If the project uses one protocol only, defer this gate - use the matching per-tool SKILL.md's "CI integration" section directly.

Step 1 - Identify your sources

For each protocol the deployment consumes, pick the tool from the matching plugin skill and ensure its output is captured as a CI artifact (always with if: always() on the upload):

ToolSkillArtifact
Pactpact-contract-testingcan-i-deploy exit code + matrix
oasdiffopenapi-contract-diff--format json array
GraphQLgraphql-schema-regressiongrep-parsed text or JSON action
Protobufprotobuf-compat-checking--error-format json array

Step 2 - Define the unified record

Flatten every tool's per-finding output into one shape:

{
  "tool":     "oasdiff",
  "protocol": "openapi",
  "finding":  "response-success-status-removed",
  "severity": "blocker",
  "subject":  "GET /pets",
  "message":  "the success response status '200' was removed",
  "ratchet":  false,
  "owner":    "@api-platform"
}
FieldSource
toolpact / oasdiff / graphql-inspector / buf.
protocolpact / openapi / graphql / protobuf.
findingtool-specific rule ID (oasdiff id, graphql-inspector change type, buf rule like FIELD_NO_DELETE).
severitynormalized - blocker / warn / info.
subjectendpoint / type / message + field that's affected.
messagehuman-readable rationale (tool's own message).
ratchetoptional - true if grandfathered; false blocks if severity is blocker.
owneroptional - team/handle responsible for the surface.

Severity normalization

ToolTool severityNormalized
Pactcan-i-deploy exit 1blocker
oasdiffERRblocker
oasdiffWARNwarn
oasdiffINFOinfo
graphql-inspectorBreakingblocker
graphql-inspectorDangerouswarn
graphql-inspectorNon-breakinginfo
buf breakingany rule under active categoryblocker

buf breaking doesn't classify by severity - every violation is treated as a blocker. To downgrade, exclude rules in buf.yaml rather than at the gate level (see protobuf-compat-checking).

Step 3 - Apply the gate decision rule

Pseudocode:

def gate_decision(records, *, allow_warn=True):
    blockers = [
        r for r in records
        if r["severity"] == "blocker" and not r.get("ratchet", False)
    ]
    warnings = [r for r in records if r["severity"] == "warn"]
    return {
        "verdict": "no-go" if blockers else "go",
        "blocker_count": len(blockers),
        "warning_count": len(warnings),
        "blockers": blockers,
        "warnings": warnings,
    }

Default behavior is strict-but-warn-tolerant: any non-ratcheted blocker fails the gate; warnings show up in the report but don't block. For stricter projects, set allow_warn=False.

Step 4 - Emit the artifact

Markdown summary suitable for $GITHUB_STEP_SUMMARY:

# Contract Compatibility Gate — verdict: NO-GO

**Blockers: 2**

| Tool      | Protocol | Finding                          | Subject                | Message                                       |
|-----------|----------|----------------------------------|------------------------|-----------------------------------------------|
| oasdiff   | openapi  | response-success-status-removed  | GET /pets              | success response status '200' was removed     |
| buf       | protobuf | FIELD_NO_DELETE                  | orders/v1.Order.amount | Field "amount" was deleted (active in v1).    |

**Warnings: 1**

| Tool              | Protocol | Finding              | Subject               | Message                              |
|-------------------|----------|----------------------|-----------------------|--------------------------------------|
| graphql-inspector | graphql  | Dangerous            | Pet.legacyTag (added union member) | adding a union member could break enums |

Plus a JSON sibling for downstream consumers:

{
  "verdict": "no-go",
  "blocker_count": 2,
  "warning_count": 1,
  "blockers": [...],
  "warnings": [...]
}

A no-go verdict exits non-zero so CI halts.

Worked example: minimal Python implementation

# scripts/run_contract_gate.py
import json, subprocess, sys
from pathlib import Path

records = []

# 1. Pact: trust can-i-deploy's exit code
pact_proc = subprocess.run(
    ["pact-broker", "can-i-deploy",
     "--pacticipant", "web-app",
     "--version", os.environ["GITHUB_SHA"],
     "--to-environment", "production"],
    capture_output=True, text=True,
)
if pact_proc.returncode != 0:
    records.append({
        "tool": "pact", "protocol": "pact",
        "finding": "can-i-deploy-no",
        "severity": "blocker",
        "subject": "web-app -> production",
        "message": pact_proc.stdout.strip().splitlines()[-1] if pact_proc.stdout else "no",
    })

# 2. oasdiff
if Path("oasdiff.json").exists():
    for f in json.loads(Path("oasdiff.json").read_text()):
        sev_map = {3: "blocker", 2: "warn", 1: "info"}
        records.append({
            "tool": "oasdiff", "protocol": "openapi",
            "finding": f["id"],
            "severity": sev_map.get(f["level"], "info"),
            "subject": f"{f['operation']} {f['path']}",
            "message": f["text"],
        })

# 3. graphql-inspector  (parsed from text or JSON action)
# 4. buf breaking      (--error-format json)
# ... (omitted for brevity; same shape)

# Apply gate
blockers = [r for r in records if r["severity"] == "blocker"]
verdict = "no-go" if blockers else "go"

print(f"# Contract Compatibility Gate — verdict: {verdict.upper()}")
for r in blockers:
    print(f"- {r['tool']} :: {r['subject']} :: {r['finding']} ({r['message']})")

sys.exit(0 if verdict == "go" else 1)

Wire into CI after every protocol step has produced its artifact:

- run: pact-broker can-i-deploy ... || true       # don't fail; let gate decide
- run: oasdiff breaking ... --format json > oasdiff.json || true
- run: graphql-inspector diff ... --json > gqi.json || true
- run: buf breaking ... --error-format json > buf.json || true

- name: Contract compatibility gate
  run: python scripts/run_contract_gate.py

The || true lets each tool emit its artifact even on failure; the final gate is the single source of CI truth.

References