webhook-delivery-tester

Overview

The Standard Webhooks spec (standardwebhooks.com (opens in new window)) formalizes the signing, retry, and replay patterns most production systems converged on. This skill covers tests for both sides:

• Sender - your service emits webhooks to customers (Stripe-style outbound webhook).
• Receiver - your service receives webhooks from a vendor (handler for Stripe / Twilio / SendGrid / GitHub events).

When to use

• The repo emits or receives webhooks.
• A regression suite needs to verify signature handling, retry semantics, replay-window enforcement.
• Compliance review (SOC 2, security audit) requires evidence of signature verification.
• The team adopts Standard Webhooks per stdwh (opens in new window).

Step 1 - Sender vs receiver test patterns

Step 2 - Sender: signature signing

Per Standard Webhooks (stdwh (opens in new window)) the canonical signature scheme:

signature = HMAC-SHA256(secret, "{webhook_id}.{timestamp}.{payload}")

The signature accompanies the payload via the webhook-signature header (with v1, prefix for the version):

webhook-id: msg_2KkD9ApUQYKLn9ouQOFKjC
webhook-timestamp: 1714838400
webhook-signature: v1,g0hM9SsE+OTPJTGt/tmIKtSyZlE3uFJELVlNIOLJ1OE=

Sender-side test:

import hmac, hashlib, base64

def sign_webhook(secret, webhook_id, timestamp, payload):
    to_sign = f"{webhook_id}.{timestamp}.{payload}".encode()
    sig = hmac.new(
        base64.b64decode(secret),
        to_sign,
        hashlib.sha256,
    ).digest()
    return f"v1,{base64.b64encode(sig).decode()}"

def test_outbound_webhook_signed_correctly():
    payload = '{"type":"order.created","data":{"id":123}}'
    webhook_id = "msg_test"
    timestamp = "1714838400"
    expected_sig = sign_webhook(SECRET, webhook_id, timestamp, payload)

    sent_request = capture_outbound_webhook()
    assert sent_request["headers"]["webhook-id"] == webhook_id
    assert sent_request["headers"]["webhook-timestamp"] == timestamp
    assert sent_request["headers"]["webhook-signature"] == expected_sig

Step 3 - Sender: retry semantics

Per stdwh (opens in new window) the canonical retry pattern is exponential backoff with jitter, capped at N attempts, then dead-letter. The typical delay schedule is in references/vendor-payloads-and-retries.md.

Sender-side test:

def test_webhook_retries_on_5xx(mock_receiver):
    mock_receiver.return_status(503)   # first 3 attempts fail
    mock_receiver.return_status_after_n(4, 200)   # 4th succeeds

    send_webhook(...)

    # Assert delivery succeeded after retries
    assert mock_receiver.attempt_count == 4
    # Assert backoff between attempts (mock can record timestamps)
    deltas = mock_receiver.attempt_deltas()
    assert deltas[1] >= 5   # at least 5s between attempt 1 and 2
    assert deltas[2] >= 5 * 60   # 5 min between 2 and 3

For dead-letter: assert that after max attempts, the webhook is recorded in a dead-letter store + not retried.

Step 4 - Receiver: signature verification

Standard Webhooks signature verification per stdwh (opens in new window):

1. Reconstruct the to-sign string: {webhook-id}.{webhook-timestamp}.{payload}
2. Compute expected signature with the shared secret.
3. Constant-time compare with the webhook-signature header value.
4. Reject if mismatch.

Receiver-side test:

def test_receiver_rejects_invalid_signature(client):
    response = client.post(
        "/webhooks/orders",
        headers={
            "webhook-id": "msg_1",
            "webhook-timestamp": str(int(time.time())),
            "webhook-signature": "v1,deliberately-wrong",
        },
        data='{"type":"order.created"}',
    )
    assert response.status_code == 400

def test_receiver_accepts_valid_signature(client):
    payload = '{"type":"order.created"}'
    timestamp = str(int(time.time()))
    sig = sign_webhook(SECRET, "msg_1", timestamp, payload)

    response = client.post(
        "/webhooks/orders",
        headers={
            "webhook-id": "msg_1",
            "webhook-timestamp": timestamp,
            "webhook-signature": sig,
        },
        data=payload,
    )
    assert response.status_code == 200

Step 5 - Receiver: replay-window defense

A captured signed payload should NOT be re-replayable indefinitely. Per stdwh (opens in new window) the receiver should reject payloads with timestamps outside a short window (typically 5 minutes).

def test_receiver_rejects_stale_timestamp(client):
    stale_timestamp = str(int(time.time()) - 600)   # 10 min ago
    payload = '{"type":"order.created"}'
    sig = sign_webhook(SECRET, "msg_1", stale_timestamp, payload)

    response = client.post(
        "/webhooks/orders",
        headers={
            "webhook-id": "msg_1",
            "webhook-timestamp": stale_timestamp,
            "webhook-signature": sig,
        },
        data=payload,
    )
    assert response.status_code == 400

If receiver doesn't enforce this, mark critical: replay vulnerable. The harder receiver attack cases - tampered payloads, future-dated timestamps, key rotation, and a capture-and-replay framework with runtime-signed fixtures - are in references/inbound-replay.md.

Step 6 - Receiver: idempotent processing

Webhooks are sent at-least-once (vendor retries on 5xx); receiver must be idempotent. Cross-ref idempotency-test-author (in the qa-async-jobs plugin):

def test_receiver_idempotent_via_webhook_id(client):
    payload = '{"type":"order.created","data":{"id":123}}'
    webhook_id = "msg_unique"
    sig, ts = sign_for_now(payload, webhook_id)

    # Send twice (simulating vendor redelivery)
    response1 = client.post("/webhooks/orders", headers=..., data=payload)
    response2 = client.post("/webhooks/orders", headers=..., data=payload)

    assert response1.status_code == 200
    assert response2.status_code == 200   # both OK, but only one side-effect
    assert Order.objects.filter(external_id=123).count() == 1

Step 7 - Per-vendor sample payloads

For receiver tests of specific vendors, use the vendor's official sample payloads (NOT hand-rolled) - making up payloads risks field-name drift. The per-vendor documentation links are in references/vendor-payloads-and-retries.md.

Step 8 - Ordering guarantees

Webhooks are typically NOT ordered (concurrent retries → out-of-order delivery). If your handler relies on order (e.g., processing order.created before order.updated), tests should:

def test_handler_handles_out_of_order_events(client):
    # Send "updated" event BEFORE "created"
    send_webhook(client, type="order.updated", id=123, status="shipped")
    send_webhook(client, type="order.created", id=123, status="placed")

    # Handler should reconcile via fetch-from-vendor + apply latest state
    order = Order.objects.get(external_id=123)
    assert order.status == "shipped"   # latest wins

If your handler can't survive out-of-order, mark critical - production will encounter this.

Step 9 - End-to-end test recipe

For sender:

1. ✅ Payload shape matches spec (Step 1)
2. ✅ Signature correctly computed (Step 2)
3. ✅ Retry on 5xx with backoff (Step 3)
4. ✅ Dead-letter after max attempts (Step 3)

For receiver:

1. ✅ Reject invalid signature (Step 4)
2. ✅ Accept valid signature (Step 4)
3. ✅ Reject stale timestamp (Step 5)
4. ✅ Idempotent processing on retry (Step 6)
5. ✅ Per-vendor sample payload tests (Step 7)
6. ✅ Out-of-order handling if applicable (Step 8)

Anti-patterns

Limitations

• This is a build-an-X workflow. Tests use the application's HTTP server + an HMAC library.
• Standard Webhooks adoption varies - many vendors use legacy signature schemes (X-Hub-Signature from GitHub, Stripe-Signature from Stripe). Adapt the verification logic per vendor; the workflow is the same.
• Signature secret rotation is out of scope; pair with a secrets-management workflow.
• For vendors that don't sign (legacy / cheap webhooks), IP-allowlisting is the only protection; not addressed here.

References

• Inbound capture-and-replay hardening (tamper / future-timestamp / key-rotation cases, sanitized captures): references/inbound-replay.md
• stdwh (opens in new window) - Standard Webhooks specification
• IETF RFC 2104 - HMAC: Keyed-Hashing for Message Authentication
• stripe.com/docs/webhooks - Stripe webhooks reference (de-facto standard for many patterns)
• docs.github.com/en/webhooks - GitHub webhooks reference
• idempotency-test-author - companion: receivers must be idempotent (cross-plugin)
• mailpit-testing (email flows in its references), sms-test-author - sister channels (bounce/complaint webhooks + STOP-keyword webhooks reuse these patterns)