Testland
Browse all skills & agents

webhook-replay-tests

Tests inbound webhook receivers for replay-attack resistance: capture incoming webhook payloads + headers, replay against the receiver under test, validate the Standard Webhooks signature scheme (svix-id + svix-timestamp + svix-signature, HMAC-SHA256 over `{id}.{timestamp}.{payload}`), svix-id idempotency dedup, and 5-minute timestamp-window enforcement by signing fixtures at runtime. Does NOT cover outbound delivery, retry-on-5xx, or failure-event exhaustion - those belong to an outbound webhook delivery harness. Use when testing the receiving side of a webhook integration.

Install with skills.sh (any agent)

npx skills add testland/qa --skill webhook-replay-tests
View source

webhook-replay-tests

Per the Standard Webhooks spec (opens in new window), "every webhook implementation needs to protect themselves and their users from SSRF, spoofing, and replay attacks." Replay tests verify the receiver enforces the signature, the timestamp window, and idempotency-key dedup.

When to use

  • Building or auditing a webhook receiver (Stripe / GitHub / Twilio / Slack inbound).
  • Pre-merge gate: signature verification works on the canary receiver before flipping traffic.
  • Outage retro: was the production failure caused by a webhook storm or by a real bug? Replay the captured payloads.

Step 1 - Capture-and-replay framework structure

tests/webhook-replay/
├── fixtures/
│   ├── stripe-charge-succeeded.json    # full request body
│   ├── stripe-charge-succeeded.headers.json   # incl. svix-* headers
│   └── github-pr-opened.json
├── replay.py                            # replay loop
└── conftest.py                          # signing helpers

Step 2 - Standard Webhooks signature scheme

Per the Standard Webhooks spec (opens in new window), required headers:

HeaderMeaning
svix-idUnique webhook identifier
svix-timestampUnix timestamp (seconds)
svix-signaturev1,<base64-hmac-sha256> (one or more, space-separated)

Signature input: HMAC-SHA256 over {id}.{timestamp}.{payload} with the shared secret as key.

Step 3 - Sign a fixture for replay

import hmac, hashlib, base64, time, json

def sign_webhook(secret_b64: str, msg_id: str, payload: bytes,
                  timestamp: int | None = None) -> dict[str, str]:
    timestamp = timestamp or int(time.time())
    secret = base64.b64decode(secret_b64.removeprefix("whsec_"))
    signed_payload = f"{msg_id}.{timestamp}.".encode() + payload
    sig = base64.b64encode(hmac.new(secret, signed_payload, hashlib.sha256).digest()).decode()
    return {
        "svix-id": msg_id,
        "svix-timestamp": str(timestamp),
        "svix-signature": f"v1,{sig}",
        "Content-Type": "application/json",
    }

Step 4 - Replay valid signed webhook (positive)

import requests
from pathlib import Path

def test_valid_signed_webhook_accepted():
    payload = Path("fixtures/stripe-charge-succeeded.json").read_bytes()
    headers = sign_webhook(
        secret_b64="whsec_<test-secret>",
        msg_id="msg_test_1",
        payload=payload,
    )
    resp = requests.post("http://localhost:8080/webhooks/stripe",
                          data=payload, headers=headers)
    assert resp.status_code == 200

Step 5 - Replay-attack tests (negative)

Per the Standard Webhooks spec (opens in new window), a 5-minute window prevents replay:

def test_old_timestamp_rejected():
    payload = b'{"event":"x"}'
    old_ts = int(time.time()) - 600  # 10 min ago, > 5 min window
    headers = sign_webhook(
        secret_b64="whsec_<test-secret>",
        msg_id="msg_test_old",
        payload=payload,
        timestamp=old_ts,
    )
    resp = requests.post("http://localhost:8080/webhooks/stripe",
                          data=payload, headers=headers)
    assert resp.status_code in (400, 401)
def test_future_timestamp_rejected():
    payload = b'{"event":"x"}'
    future_ts = int(time.time()) + 600
    headers = sign_webhook("whsec_<test-secret>", "msg_test_future",
                            payload, timestamp=future_ts)
    resp = requests.post("http://localhost:8080/webhooks/stripe",
                          data=payload, headers=headers)
    assert resp.status_code in (400, 401)

Step 6 - Tampered-payload test

def test_tampered_payload_rejected():
    payload = b'{"amount":100}'
    headers = sign_webhook("whsec_<test-secret>", "msg_test_tamper", payload)

    # Tamper after signing
    tampered = b'{"amount":1000000}'
    resp = requests.post("http://localhost:8080/webhooks/stripe",
                          data=tampered, headers=headers)
    assert resp.status_code in (400, 401)

Step 7 - Idempotency / dedup test

Per the Standard Webhooks spec (opens in new window), "idempotency-key behavior" handles duplicates gracefully. The receiver should treat a second delivery of the same svix-id as a no-op (or a 200 with no side effects).

def test_duplicate_svix_id_does_not_duplicate_side_effect():
    payload = Path("fixtures/stripe-charge-succeeded.json").read_bytes()
    msg_id = "msg_dup_test_1"
    headers = sign_webhook("whsec_<test-secret>", msg_id, payload)

    # Fresh state
    reset_orders_table()

    # Deliver once
    r1 = requests.post("http://localhost:8080/webhooks/stripe",
                       data=payload, headers=headers)
    assert r1.status_code == 200
    orders_after_first = count_orders()

    # Deliver again with same svix-id
    r2 = requests.post("http://localhost:8080/webhooks/stripe",
                       data=payload, headers=headers)
    assert r2.status_code == 200  # accept, no-op
    orders_after_second = count_orders()

    assert orders_after_first == orders_after_second

Step 8 - Multi-version signature test (key rotation)

Per the Standard Webhooks spec (opens in new window), svix-signature can carry multiple versions/keys (space-separated v1,...). Receiver accepts if any key validates.

def test_accepts_during_key_rotation():
    payload = b'{"event":"x"}'
    msg_id = "msg_rotate_1"
    ts = int(time.time())

    sig_old = compute_sig(secret_b64="whsec_OLD", msg_id=msg_id,
                            payload=payload, timestamp=ts)
    sig_new = compute_sig(secret_b64="whsec_NEW", msg_id=msg_id,
                            payload=payload, timestamp=ts)

    headers = {
        "svix-id": msg_id,
        "svix-timestamp": str(ts),
        "svix-signature": f"v1,{sig_old} v1,{sig_new}",
        "Content-Type": "application/json",
    }
    resp = requests.post("http://localhost:8080/webhooks/stripe",
                          data=payload, headers=headers)
    assert resp.status_code == 200

Step 9 - Capture from production (responsibly)

For captured payloads, sanitize before committing:

def sanitize_capture(payload: dict) -> dict:
    SENSITIVE_KEYS = {"email", "phone", "ssn", "card", "address"}
    def walk(node):
        if isinstance(node, dict):
            return {k: ("***" if k.lower() in SENSITIVE_KEYS else walk(v))
                     for k, v in node.items()}
        if isinstance(node, list):
            return [walk(x) for x in node]
        return node
    return walk(payload)

Anti-patterns

Anti-patternWhy it failsFix
Skip signature test in dev (mock the verifier)Prod-only signature bug shipsUse the same verifier in test as prod (Step 4)
Hard-code timestamps in fixturesOld fixtures fail Step 5 windowed-replay protectionSign at test runtime (Step 3)
No idempotency testDuplicate webhook deliveries cause double-chargesStep 7 mandatory
Commit raw production payloadsPII leak in repoSanitize before commit (Step 9)
Use single key, no rotation pathForced re-signing at rotation; outage riskMulti-key acceptance (Step 8)

Limitations

  • Standard Webhooks is an emerging spec; some senders (Stripe, GitHub) use proprietary header names. The svix-* / Stripe-Signature / X-Hub-Signature-256 schemes share the HMAC-SHA256 pattern but differ in headers + signature serialization.
  • 5-minute window is the convention; some receivers tighten or widen - verify against the spec your sender publishes.
  • Replay tests don't catch live-network issues (TLS errors, DNS). Pair with webhook-delivery-tester for delivery
    • retry policy tests.

References

  • Standard Webhooks spec (opens in new window) - signature scheme, replay window, idempotency
  • webhook-delivery-tester (in the qa-notifications plugin) - sister skill for delivery + retry on the sender side
  • mqtt-tests, websocket-tests - alternative realtime transports

Related skills

grpc-streaming-tests

Test gRPC streaming RPCs - Server-streaming (server returns sequence), Client-streaming (client sends sequence), Bidirectional (both sides stream independently). Cover deadline + cancellation + flow control + status codes (CANCELLED, DEADLINE_EXCEEDED) + metadata. Use ghz for load, grpcurl for ad-hoc, language-native test stubs for unit/integration. Use when a service exposes server-, client-, or bidirectional-streaming RPCs and deadline, cancellation, or partial-stream status-code behavior is unverified.

mqtt-tests

Test MQTT v5.0 with Mosquitto broker in CI + paho-mqtt clients - QoS 0 / 1 / 2 delivery semantics, retained messages, Last Will and Testament (LWT), shared subscriptions ($share/group/topic), $SYS topic introspection. Critical for IoT, embedded, and M2M systems where wire-level guarantees matter. Use when a product speaks MQTT on the wire and QoS 1 / 2 redelivery, retained-message state, or LWT behavior needs a broker-backed test - including smoke-testing a new broker auth / ACL / persistence config.

server-sent-events-tests

Test Server-Sent Events (SSE) flows, one-way server-to-client push only (not bidirectional, use websocket-tests for client-to-server messaging): `EventSource` API on the browser side (`onmessage`, `onerror`, `readyState` 0/1/2), event stream format (`data:`, `event:`, `id:`, `retry:`), `Last-Event-ID` reconnect-with-replay header, content-type `text/event-stream`, and HTTP/1.1 connection-pool limits. Use Playwright for browser-side, raw HTTP client for server-side stream tests. Use when a feature pushes updates over `text/event-stream` and the reconnect interval, `Last-Event-ID` replay, or per-origin connection ceiling has no coverage.

sse-load-tests

Load-tests SSE endpoints at scale with k6 - measures concurrent-stream capacity, connection churn, and server memory pressure. Covers the HTTP/1.1 6-connection-per-origin browser ceiling vs HTTP/2 multiplexing, a custom k6 SSE client built on ReadableStream, and threshold gates for TTFB and data throughput. Use when validating whether a server can sustain N concurrent EventSource connections without connection starvation or memory growth.

stomp-amqp-tests

Tests STOMP over WebSocket (Spring, ActiveMQ, RabbitMQ Web STOMP) and AMQP 0-9-1 (RabbitMQ Java client) - frame connect/subscribe/send/ack sequences, ack modes (auto/client/client-individual), exchange and queue declarations, binding routing, Testcontainers RabbitMQ broker, and delivery assertion. Use when validating enterprise Spring or RabbitMQ messaging stacks before deploy.

websocket-tests

Test WebSocket protocol behavior - opening handshake (HTTP Upgrade with Sec-WebSocket-Key + Sec-WebSocket-Version: 13), control frames (ping 0x9 / pong 0xA / close 0x8), close-frame status codes (1000 normal, 1001 going-away, 1006 abnormal, 1011 server error), subprotocol negotiation, backpressure, and reconnect with jitter. Works with ws (Node), websockets (Python), or Playwright frame inspection per language. Use when a feature holds a long-lived WebSocket open and reconnect, close-code, or backpressure behavior is unverified.