Testland
Browse all skills & agents

grpc-streaming-test-author

The single gRPC-streaming test home: builds streaming-RPC test suites from a proto definition. Classifies each RPC by pattern (unary, server-streaming, client-streaming, bidi), then emits the required categories per pattern - ordering preservation, completion semantics (server close after stream end, client half-close), cancellation, deadline handling, partial-stream failure. Produces skeletons for Go (bufconn + Send/Recv), Python (iterators), JVM (StreamObserver), Node (call.write/end); carries the 17-code gRPC status catalog (retry semantics per AIP-194, grpc-gateway HTTP mapping) in references/status-codes.md and the wire-level / live-server streaming patterns (deadline propagation, server-side cancellation, metadata, ghz load) in references/wire-level-testing.md. Use when adding tests for a new or existing streaming RPC, auditing a suite for uncovered categories, or asserting status-code behavior. Different test surface from grpc-mock (the in-process harness itself).

Install with skills.sh (any agent)

npx skills add testland/qa --skill grpc-streaming-test-author
View source

grpc-streaming-test-author

Overview

Streaming RPCs are where gRPC clients and servers most often diverge from the proto contract - message ordering, completion signalling, cancellation, and partial-failure semantics are all testable surfaces. gRPC defines four streaming patterns - unary, server-streaming, client-streaming, bidirectional - per grpc.io/docs/what-is-grpc/core-concepts/ (opens in new window); Step 1 classifies each RPC by pattern.

This skill walks through producing a comprehensive test suite from the proto file. It composes grpc-mock for the test harness and references/status-codes.md for status-code assertions; live-server (non-bufconn) patterns are in references/wire-level-testing.md.

When to use

  • Adding tests for a new streaming RPC.
  • Auditing an existing streaming-RPC test suite - is each pattern's required category covered?
  • Investigating a streaming-RPC bug - minimal repro from the test matrix.
  • PR review of changes to streaming RPCs (the proto-evolution rules on stream changes are subtle - per buf-cli-lint-breaking-build references/versioning-strategy.md).

Step 1 - Classify each RPC

Walk the .proto file and tabulate:

service Chat {
  rpc Send(Message) returns (Ack);                         // unary
  rpc Subscribe(SubReq) returns (stream Event);            // server streaming
  rpc Upload(stream Chunk) returns (UploadResult);         // client streaming
  rpc Conversation(stream Message) returns (stream Reply); // bidi
}
RPCPatternRequired test categories
Sendunarysuccess, every status code per references/status-codes.md
Subscribeserver-streamsuccess, ordering, server-side close after N messages, server-side mid-stream error, client-side cancel mid-stream, deadline-exceeded mid-stream
Uploadclient-streamsuccess, server completes before client finishes, server-side error mid-upload, client-side cancel before send, empty stream
Conversationbidisuccess, ordering per direction, client closes send while still receiving, server closes send while still receiving, both close, deadline mid-conversation, error mid-conversation

Step 2 - Test categories per pattern

Unary (covered for completeness)

def test_send_returns_ok(stub):
    resp = stub.Send(Message(body="hi"))
    assert resp.ok

def test_send_returns_invalid_argument(stub):
    with pytest.raises(grpc.RpcError) as exc:
        stub.Send(Message(body=""))
    assert exc.value.code() == grpc.StatusCode.INVALID_ARGUMENT

Server-streaming

Ordering preservation - per gRPC docs: "gRPC guarantees message ordering within an individual RPC call."

def test_subscribe_preserves_ordering(stub_with_fake):
    stub_with_fake.fake_response_stream = [
        Event(seq=1), Event(seq=2), Event(seq=3),
    ]
    received = list(stub_with_fake.Subscribe(SubReq()))
    assert [e.seq for e in received] == [1, 2, 3]

Server closes after N messages - completion is the finalisation signal:

def test_subscribe_completes_after_finite_stream(stub_with_fake):
    stub_with_fake.fake_response_stream = [Event(seq=1), Event(seq=2)]
    events = list(stub_with_fake.Subscribe(SubReq()))
    # If list(...) terminates, the server-side close was signalled.
    assert len(events) == 2

Server-side error mid-stream:

def test_subscribe_propagates_error_mid_stream(stub_with_fake):
    stub_with_fake.fake_response_stream = [Event(seq=1)]
    stub_with_fake.fake_post_yield_status = grpc.StatusCode.INTERNAL
    events_iter = stub_with_fake.Subscribe(SubReq())
    # Receive first event OK
    next(events_iter)
    # Subsequent receive fails with INTERNAL per references/status-codes.md
    with pytest.raises(grpc.RpcError) as exc:
        next(events_iter)
    assert exc.value.code() == grpc.StatusCode.INTERNAL

Client cancel mid-stream:

def test_client_cancel_mid_stream(stub_with_fake):
    stub_with_fake.fake_response_stream = [Event(seq=i) for i in range(100)]
    events_iter = stub_with_fake.Subscribe(SubReq())
    next(events_iter)  # receive one
    events_iter.cancel()
    with pytest.raises(grpc.RpcError) as exc:
        next(events_iter)
    assert exc.value.code() == grpc.StatusCode.CANCELLED

Deadline-exceeded mid-stream:

def test_subscribe_deadline_exceeded(stub_with_slow_fake):
    events_iter = stub_with_slow_fake.Subscribe(SubReq(), timeout=0.1)
    with pytest.raises(grpc.RpcError) as exc:
        for _ in events_iter:
            pass
    assert exc.value.code() == grpc.StatusCode.DEADLINE_EXCEEDED

Client-streaming

Categories: success, server completes before client finishes (further Send yields io.EOF), server-side error mid-upload, client-side cancel before send, empty stream. Go bufconn skeletons: references/test-skeletons.md.

Bidirectional

The two streams operate independently, so test each direction plus a concurrent send/recv case. Categories: ordering per direction, client closes send while still receiving (CloseSend), server closes send while still receiving, both close, deadline mid-conversation, error mid-conversation. Go skeletons: references/test-skeletons.md.

Step 3 - Coverage matrix

For each streaming RPC, generate the matrix:

                              Send  Subscribe  Upload  Conversation
                              ----  ---------  ------  ------------
success                        X       X        X         X
ordering preserved             -       X        X         X (both)
N msgs then close              -       X        X         X
mid-stream server error        -       X        X         X
deadline exceeded              X       X        X         X
client cancel                  X       X        X         X
server cancel                  -       X        X         X
empty stream                   -       X        X         X
client half-close              -       -        -         X
server half-close              -       -        -         X

Empty cells in covered patterns = coverage gap. PR must justify or add a test.

Step 4 - Pick the test harness

Per grpc-mock:

LanguageHarness
Gobufconn in-process server + t.Cleanup
Pythonpytest fixture with [::]:0 port + iterator-based stream API
JVMInProcessServerBuilder + StreamObserver
Node@grpc/grpc-js server with bindAsync("127.0.0.1:0") + call.write / call.end

Don't use mockgen / interface-mocks for streaming - they skip the marshalling + ordering guarantees that the streaming contract depends on.

Step 5 - Emit the test directory

tests/grpc-streaming/
  __init__.py
  conftest.py                   # shared fixtures (in-process server)
  test_subscribe.py             # server-streaming
  test_upload.py                # client-streaming
  test_conversation.py          # bidi
  test_send.py                  # unary (sibling)
  README.md                     # the coverage matrix

The README.md should document the matrix from Step 3 so reviewers and new contributors can see at a glance what is and isn't covered.

Streaming evolution - version-safety reminder

Per the versioning-strategy reference in buf-cli-lint-breaking-build, changing an RPC's streaming pattern is always breaking:

service Chat {
-  rpc Subscribe(SubReq) returns (Event);
+  rpc Subscribe(SubReq) returns (stream Event);
}

The wire format differs, so old clients won't parse the new response. Verify breaking-change detection via buf-cli-lint-breaking-build catches this with FILE or PACKAGE category.

Anti-patterns

Anti-patternWhy it failsFix
Single happy-path test per streaming RPCMisses ordering / cancellation / deadline categoriesUse Step 2's per-pattern matrix
Test asserts on the last message onlyMisses ordering bugs in earlier messagesCollect entire stream, assert on the full sequence
time.sleep to wait for server messagesRace-prone; flakes in CIUse channel close / iterator exhaustion as completion signal
Cancellation test sleeps then cancelsRace: server may finish before cancelInject a controllable blocker in the fake server
No deadline-exceeded testProduction deadlines surface in mid-streamAlways include - per references/status-codes.md, DEADLINE_EXCEEDED is its own code
Mock at interface level for streamingSkips marshalling + orderingIn-process server only for streaming
Bidi test where client and server are deterministic-interleavedMisses race conditions inherent in "operate independently"One test per direction + one test with concurrent send/recv
Don't CloseSend() in client-streaming testsServer waits foreverAlways close the send side explicitly

Limitations

  • Wire-level fault injection. Partial-byte cutoffs and middlebox-induced disconnects aren't reachable through bufconn / InProcessServer. For these, use toxiproxy + a real server; live-server patterns in references/wire-level-testing.md.
  • Backpressure semantics differ per stack. Go bufconn buffer size affects when Send blocks; Java StreamObserver is push-based; Python is iterator-pull. Tests are stack-specific.
  • HTTP/2 flow control isn't exercised. In-process transports skip H2 framing; flow-control bugs need real transports.
  • Doesn't test contract compliance. Mocks reflect what this test expects. Real-server contract tests are separate; see protobuf-compat-checking (in the qa-contract-testing plugin).
  • Concurrent-client stress tests not in scope. For that see ghz-load (unary load) or hand-rolled multi-client streaming harnesses.

References

gRPC status codes, metadata, and load testing

View source (opens in new window)

gRPC status codes, metadata, and load testing

Cross-cutting call semantics beyond the four streaming patterns: the full status-code matrix, request/response metadata round-trip, and load testing with ghz.

Status codes

CodeWhen
OKSuccess
CANCELLEDClient cancelled
DEADLINE_EXCEEDEDDeadline elapsed
INVALID_ARGUMENTClient error in request
UNAUTHENTICATEDNo / bad credentials
PERMISSION_DENIEDAuthenticated but not authorized
RESOURCE_EXHAUSTEDQuota / rate limit
INTERNALServer bug
UNAVAILABLEServer transient unreachable (clients should retry)

Test the error path returns the right code, not just "an error":

def test_invalid_argument_returns_correct_code():
    with grpc.insecure_channel("localhost:50051") as ch:
        stub = OrdersStub(ch)
        with pytest.raises(grpc.RpcError) as exc:
            stub.CreateOrder(OrderRequest(item_count=-1))
        assert exc.value.code() == grpc.StatusCode.INVALID_ARGUMENT

Metadata

Per the gRPC core concepts docs (opens in new window), metadata is "key-value pairs" case-insensitive ASCII keys; binary values use -bin suffix.

def test_request_metadata_round_trip():
    with grpc.insecure_channel("localhost:50051") as ch:
        stub = OrdersStub(ch)
        metadata = (("x-trace-id", "abc123"),)
        resp, call = stub.CreateOrder.with_call(OrderRequest(), metadata=metadata)

        # Server reflects request-id in response trailing metadata
        trailing = call.trailing_metadata()
        assert ("x-trace-id-echo", "abc123") in trailing

Load test with ghz

ghz \
  --insecure \
  --proto orders.proto \
  --call orders.Orders/CreateOrder \
  -d '{"item_count":1}' \
  -c 50 \
  -n 10000 \
  localhost:50051

Reports RPS, p50/p95/p99 latency. For streaming RPCs use --stream-call-count flag (consult ghz docs).

gRPC status codes: semantics, retries, HTTP mapping

View source (opens in new window)

gRPC status codes: semantics, retries, HTTP mapping

gRPC defines 17 standard status codes (per grpc.io/docs/guides/status-codes/ (opens in new window)) that every implementation respects. They are the wire-level error vocabulary; using the wrong code breaks retry behaviour, HTTP gateway translation, and observability dashboards.

Three things to get right:

  1. The semantic meaning - OK..UNAUTHENTICATED have specific definitions; picking the wrong one mis-signals to clients.
  2. The retry behaviour - per AIP-194 (opens in new window), only UNAVAILABLE is generally safe to auto-retry.
  3. The HTTP mapping - per grpc-gateway runtime/errors.go (opens in new window), FAILED_PRECONDITION → 400 Bad Request (NOT 412 "Precondition Failed" despite the name).

This reference serves client-test authors, server implementers, and gRPC-to-HTTP gateway configurators.

When to use

  • Designing the error vocabulary of a new gRPC service.
  • Writing client-test assertions: which status should the test expect?
  • Configuring a retry policy in the gRPC client.
  • Mapping gRPC errors to HTTP in a grpc-gateway setup.
  • PR review - is this status.Errorf call using the right code?

The canonical 17 codes

Per grpc.io/docs/guides/status-codes/ (opens in new window):

#CodeDefinition (gRPC docs)Typical example
0OK"Not an error; returned on success."Successful RPC
1CANCELLED"The operation was cancelled, typically by the caller."Client terminates stream
2UNKNOWN"Unknown error... errors raised by APIs that do not return enough error information."Wrapping a non-gRPC error
3INVALID_ARGUMENT"The client specified an invalid argument... problematic regardless of the state of the system."Malformed request payload
4DEADLINE_EXCEEDED"The deadline expired before the operation could complete... even if the operation has completed successfully."Server slow + deadline expired
5NOT_FOUND"Some requested entity (e.g., file or directory) was not found."Resource doesn't exist
6ALREADY_EXISTS"The entity that a client attempted to create already exists."Duplicate unique key
7PERMISSION_DENIED"The caller does not have permission to execute the specified operation."Authenticated but unauthorised
8RESOURCE_EXHAUSTED"Some resource has been exhausted, perhaps a per-user quota."Rate limit hit
9FAILED_PRECONDITION"System is not in a state required for the operation's execution... client should not retry until the system state has been explicitly fixed."Delete non-empty bucket
10ABORTED"The operation was aborted, typically due to a concurrency issue such as a sequencer check failure or transaction abort."Optimistic-locking conflict
11OUT_OF_RANGE"The operation was attempted past the valid range... problem that may be fixed if the system state changes."Seek past EOF
12UNIMPLEMENTED"The operation is not implemented or is not supported/enabled in this service."Method not in this version
13INTERNAL"Internal errors... invariants expected by the underlying system have been broken."Database corruption
14UNAVAILABLE"The service is currently unavailable... most likely a transient condition, which can be corrected by retrying with a backoff."Backend restarting
15DATA_LOSS"Unrecoverable data loss or corruption."Storage volume failed
16UNAUTHENTICATED"The request does not have valid authentication credentials for the operation."Missing JWT

Retry behaviour

Per AIP-194 (opens in new window):

CodeRetry?Notes
OKn/aSuccess
CANCELLEDNoClient requested cancellation; honour it
UNKNOWNNoUnsafe; retrying may compound state
INVALID_ARGUMENTNoArgument won't change
DEADLINE_EXCEEDEDNoApplication deadline must be respected
NOT_FOUNDNoRequires state change
ALREADY_EXISTSNoRequires state change
PERMISSION_DENIEDNoRequires permission change
RESOURCE_EXHAUSTEDMaybeQuota may take hours; consider billing
FAILED_PRECONDITIONNo"Client should not retry until the system state has been explicitly fixed" (gRPC docs)
ABORTEDApplication-level"Retry at the transaction level, not individual request level"
OUT_OF_RANGENoRequires state change
UNIMPLEMENTEDNoMethod doesn't exist
INTERNALNoSurface bugs immediately
UNAVAILABLEYes"The only error code explicitly recommended for automatic retry" per AIP-194
DATA_LOSSNoUnrecoverable; surface immediately
UNAUTHENTICATEDNoRe-auth first, then retry application-level

Client-side retry policy

{
  "methodConfig": [{
    "name": [{"service": "example.v1.UserService"}],
    "retryPolicy": {
      "maxAttempts": 4,
      "initialBackoff": "0.1s",
      "maxBackoff": "1s",
      "backoffMultiplier": 2,
      "retryableStatusCodes": ["UNAVAILABLE"]
    }
  }]
}

Adding RESOURCE_EXHAUSTED to retryableStatusCodes is sometimes seen but per AIP-194 has billing implications.

gRPC ↔ HTTP mapping (grpc-gateway)

Per grpc-gateway/runtime/errors.go (opens in new window):

gRPC codeHTTP statusNotes
OK200Standard success
CANCELLED499Client Closed Request (nginx-originated, non-standard)
UNKNOWN500Internal Server Error
INVALID_ARGUMENT400Bad Request
DEADLINE_EXCEEDED504Gateway Timeout
NOT_FOUND404Not Found
ALREADY_EXISTS409Conflict
PERMISSION_DENIED403Forbidden
UNAUTHENTICATED401Unauthorized
RESOURCE_EXHAUSTED429Too Many Requests
FAILED_PRECONDITION400Bad Request - NOT 412 "Precondition Failed" despite the name. grpc-gateway code comment: "deliberately doesn't translate to the similarly named '412 Precondition Failed'"
ABORTED409Conflict (concurrency)
OUT_OF_RANGE400Bad Request
UNIMPLEMENTED501Not Implemented
INTERNAL500Internal Server Error
UNAVAILABLE503Service Unavailable
DATA_LOSS500Internal Server Error

Implications for HTTP clients

  • UNAUTHENTICATED (401) vs PERMISSION_DENIED (403) - same distinction as plain HTTP.
  • INVALID_ARGUMENT, FAILED_PRECONDITION, OUT_OF_RANGE all → 400; the gRPC code carries the semantic distinction.
  • INTERNAL, UNKNOWN, DATA_LOSS all → 500.
  • HTTP 499 (Cancelled) is non-standard; some HTTP clients don't handle it.

Choosing the right code - disambiguators

The hardest pairs:

INVALID_ARGUMENT vs FAILED_PRECONDITION vs OUT_OF_RANGE

QuestionAnswer
Is the arg malformed regardless of system state?INVALID_ARGUMENT
Is the arg valid but the system isn't in the right state?FAILED_PRECONDITION
Is the arg past a valid range that may shift over time?OUT_OF_RANGE

Per gRPC docs: FAILED_PRECONDITION "client should not retry"; OUT_OF_RANGE "may be fixed if the system state changes."

FAILED_PRECONDITION vs ABORTED vs UNAVAILABLE

QuestionAnswer
Concurrency conflict (transaction abort, sequencer check)?ABORTED
System needs state change before retry?FAILED_PRECONDITION
System transient unavailability?UNAVAILABLE

Per AIP-194: ABORTED retries at the transaction level, not the request level. UNAVAILABLE retries at the request level.

NOT_FOUND vs UNIMPLEMENTED

QuestionAnswer
Resource doesn't exist at this point in time?NOT_FOUND
Method doesn't exist in this server version?UNIMPLEMENTED

UNKNOWN vs INTERNAL

QuestionAnswer
Server caught a non-gRPC error and is wrapping it?UNKNOWN
Server detected its own invariant violation?INTERNAL

UNKNOWN is for upstream noise; INTERNAL is for "I detected something wrong with me."

Tests should assert these codes

In client tests (per grpcurl-cli and language-specific clients):

import grpc
import pytest

def test_get_user_not_found_returns_not_found(stub):
    with pytest.raises(grpc.RpcError) as exc_info:
        stub.GetUser(GetUserRequest(id="nonexistent"))
    assert exc_info.value.code() == grpc.StatusCode.NOT_FOUND

def test_create_user_with_duplicate_email_returns_already_exists(stub, existing_user):
    with pytest.raises(grpc.RpcError) as exc_info:
        stub.CreateUser(CreateUserRequest(email=existing_user.email))
    assert exc_info.value.code() == grpc.StatusCode.ALREADY_EXISTS

def test_unauthenticated_call_returns_unauthenticated(stub_no_auth):
    with pytest.raises(grpc.RpcError) as exc_info:
        stub_no_auth.GetUser(GetUserRequest(id="any"))
    assert exc_info.value.code() == grpc.StatusCode.UNAUTHENTICATED

Anti-patterns

Anti-patternWhy it failsFix
Returning INTERNAL for caller-side errorsSurfaces server bugs that don't exist; alarms fireUse INVALID_ARGUMENT for bad inputs
Returning UNKNOWN everywhereNo retry semantics, no HTTP mapping clarityPick the specific code
Returning FAILED_PRECONDITION for transient unavailabilityClients don't retry → user-visible failuresUse UNAVAILABLE
Using NOT_FOUND for "you don't have permission"Information leak (tenant probes) - OR - opacity (debugging hard); document the choicePer cross-tenant-data-leak-tests (in the qa-multi-tenancy plugin) the 404-vs-403 trade-off is project policy
OK with error message in payloadBreaks every gRPC client's error handlingAlways use a non-OK status for errors
Adding all codes to retryableStatusCodesRetries non-idempotent operations; data corruptionOnly UNAVAILABLE per AIP-194 (case-by-case for others)
Treating HTTP 412 as FAILED_PRECONDITION in gatewaygrpc-gateway maps FP→400 (not 412)Verify the mapping in runtime/errors.go
Asserting on error message string in testsBreaks on i18n / wording tweaksAssert on code() only

Custom error details

For richer error info beyond the code, embed google.rpc.ErrorInfo / BadRequest / Help in status.Details:

from google.rpc import error_details_pb2, status_pb2
from grpc_status import rpc_status

def get_user_with_details(stub, user_id):
    try:
        return stub.GetUser(GetUserRequest(id=user_id))
    except grpc.RpcError as e:
        status = rpc_status.from_call(e)
        for detail in status.details:
            if detail.Is(error_details_pb2.BadRequest.DESCRIPTOR):
                bad_req = error_details_pb2.BadRequest()
                detail.Unpack(bad_req)
                for v in bad_req.field_violations:
                    print(f"{v.field}: {v.description}")

Tests should assert on detail messages where richer signal exists.

Limitations

  • No standard payload for codes. Each service defines its own "structured details" envelope (Google uses google.rpc.*; others roll their own).
  • HTTP mapping is gateway-specific. envoy, grpc-gateway, and hand-rolled adapters can differ. The grpc-gateway table above is the most common reference.
  • UNKNOWN is overused in the wild. Defensive servers default to UNKNOWN to avoid leaking internals; this defeats observability.
  • Status code metrics. Dashboards that aggregate by code lose information when servers route everything through INTERNAL.

References

Per-language streaming test skeletons

View source (opens in new window)

Per-language streaming test skeletons

Go bufconn skeletons for the client-streaming and bidirectional categories. The body of SKILL.md keeps the Python server-streaming and unary examples as the representative inline set; these are the language variants. Harness selection is in grpc-mock.

Client-streaming (Go bufconn)

Success:

func TestUpload_Success(t *testing.T) {
    fake := &fakeUploader{accept: 3}
    client := setupClient(t, fake)

    stream, err := client.Upload(context.Background())
    if err != nil { t.Fatal(err) }

    chunks := []*pb.Chunk{{Data: []byte("a")}, {Data: []byte("b")}, {Data: []byte("c")}}
    for _, c := range chunks {
        if err := stream.Send(c); err != nil { t.Fatal(err) }
    }
    result, err := stream.CloseAndRecv()
    if err != nil { t.Fatal(err) }
    if result.Bytes != 3 { t.Fatalf("got %d, want 3", result.Bytes) }
}

Server completes before client finishes - per gRPC docs the server response may arrive "typically but not necessarily after it has received all the client's messages"; further Send yields io.EOF:

func TestUpload_ServerCompletesEarly(t *testing.T) {
    fake := &fakeUploader{completeAfter: 1}
    client := setupClient(t, fake)

    stream, _ := client.Upload(context.Background())
    stream.Send(&pb.Chunk{Data: []byte("a")})

    // Sending more after server completes should yield io.EOF
    err := stream.Send(&pb.Chunk{Data: []byte("b")})
    if err != io.EOF {
        t.Fatalf("got %v, want io.EOF", err)
    }
    result, _ := stream.CloseAndRecv()
    if result.Bytes != 1 { t.Fatalf("got %d, want 1", result.Bytes) }
}

Empty stream:

func TestUpload_EmptyStream(t *testing.T) {
    client := setupClient(t, &fakeUploader{})
    stream, _ := client.Upload(context.Background())
    result, err := stream.CloseAndRecv()
    if err != nil { t.Fatal(err) }
    if result.Bytes != 0 { t.Fatalf("got %d, want 0", result.Bytes) }
}

Bidirectional (Go)

Ordering per direction:

func TestConversation_Ordering(t *testing.T) {
    fake := &fakeChatter{
        clientMsgs: []*pb.Message{},
        replies: []*pb.Reply{{Seq: 1}, {Seq: 2}, {Seq: 3}},
    }
    client := setupClient(t, fake)
    stream, _ := client.Conversation(context.Background())

    // Client sends 3 messages
    for i := 0; i < 3; i++ {
        stream.Send(&pb.Message{Seq: int32(i)})
    }
    stream.CloseSend()

    // Server sends back 3 replies in order
    var got []int32
    for {
        r, err := stream.Recv()
        if err == io.EOF { break }
        if err != nil { t.Fatal(err) }
        got = append(got, r.Seq)
    }
    want := []int32{1, 2, 3}
    if !reflect.DeepEqual(got, want) {
        t.Fatalf("got %v, want %v", got, want)
    }
}

Client closes send while still receiving (half-close):

func TestConversation_ClientHalfClose(t *testing.T) {
    // Server keeps sending after client CloseSend()
    fake := &fakeChatter{repliesAfterCloseSend: []*pb.Reply{{Seq: 99}}}
    client := setupClient(t, fake)
    stream, _ := client.Conversation(context.Background())
    stream.Send(&pb.Message{Seq: 0})
    stream.CloseSend()  // half-close: no more sends, still receiving

    r, err := stream.Recv()
    if err != nil { t.Fatal(err) }
    if r.Seq != 99 { t.Fatalf("got %d, want 99", r.Seq) }
}

Cancellation from either side:

func TestConversation_ServerSideCancel(t *testing.T) {
    fake := &fakeChatter{cancelAfterMsg: 1}
    client := setupClient(t, fake)
    stream, _ := client.Conversation(context.Background())
    stream.Send(&pb.Message{Seq: 0})

    _, err := stream.Recv()
    st, _ := status.FromError(err)
    if st.Code() != codes.Cancelled {
        t.Fatalf("got %v, want Cancelled", st.Code())
    }
}

Wire-level / live-server streaming tests

View source (opens in new window)

Wire-level / live-server streaming tests

The SKILL.md categories run against in-process harnesses (bufconn / InProcessServer); this reference covers the live-server variant - real channels against localhost:50051 - where deadline propagation, cancellation observed server-side, metadata round-trips, and ghz load behaviour are exercised over a real transport.

When to use

  • Service exposes streaming RPCs (price ticker, log tail, IoT telemetry, AI streaming inference).
  • Pre-deploy gate: deadline + cancellation propagate correctly, partial-stream errors return correct status codes.
  • Load test gate: streams handle backpressure without OOM or silent drops.

How to use

  1. Pick the test tool for the job (Step 1): native stubs for unit/integration, grpcurl for smoke, ghz for load.
  2. Prove the plumbing with a unary sanity call before touching streams (Step 2).
  3. Cover each streaming shape the service exposes - server-, client-, and bidirectional-streaming (Steps 3-5).
  4. Assert deadline propagation and client-initiated cancellation are observed server-side (Steps 6-7).
  5. Check error paths return the exact status code, not just "an error", and that request metadata round-trips (status-codes-metadata-load.md (opens in new window)).
  6. Run a ghz load pass to confirm streams handle backpressure without OOM or silent drops (same reference).
  7. Gate the suite on the anti-patterns table before merge.

Step 1 - Pick the test tool

ToolStrength
Language-native stubs (Go grpc.WithBlock(), Python grpc.aio, Java ManagedChannel)Unit/integration tests
grpcurlAd-hoc + smoke tests + scripts
ghzLoad testing + benchmarks (concurrency, RPS)
mockgrpc / mockery (Go), grpc-mock (Node)Mock server stubs in unit tests

Step 2 - Unary RPC sanity (baseline)

Per the gRPC core concepts docs (opens in new window), Unary = "single request, single response." Use this to verify the RPC plumbing before testing streams:

import grpc
from orders_pb2 import OrderRequest
from orders_pb2_grpc import OrdersStub

def test_unary_create_order():
    with grpc.insecure_channel("localhost:50051") as ch:
        stub = OrdersStub(ch)
        resp = stub.CreateOrder(OrderRequest(item_count=2), timeout=5.0)
        assert resp.order_id != ""

Step 3 - Server-streaming test

Per the gRPC core concepts docs (opens in new window), server-streaming = "client sends a request and gets a stream to read a sequence of messages back."

def test_server_streaming_price_ticker():
    with grpc.insecure_channel("localhost:50051") as ch:
        stub = PricesStub(ch)
        stream = stub.SubscribePrices(SubscribeRequest(symbol="AAPL"), timeout=10.0)
        ticks = []
        for tick in stream:
            ticks.append(tick)
            if len(ticks) >= 5:
                stream.cancel()
                break

        assert len(ticks) == 5
        assert all(t.symbol == "AAPL" for t in ticks)

Step 4 - Client-streaming test

Per the gRPC core concepts docs (opens in new window), client-streaming = "client writes a sequence of messages and sends them to the server."

def test_client_streaming_upload():
    def chunks():
        for i in range(10):
            yield UploadChunk(seq=i, data=b"x" * 1024)

    with grpc.insecure_channel("localhost:50051") as ch:
        stub = UploadsStub(ch)
        resp = stub.Upload(chunks(), timeout=10.0)

        assert resp.total_chunks == 10
        assert resp.total_bytes == 10 * 1024

Step 5 - Bidirectional streaming + ordering

Per the gRPC core concepts docs (opens in new window), bidirectional streams "operate independently" - server may emit messages before reading any client message, after, or interleaved.

import asyncio

async def test_bidi_chat():
    async def client_messages():
        for msg in ["hello", "how are you", "bye"]:
            yield ChatMessage(text=msg)
            await asyncio.sleep(0.1)

    async with grpc.aio.insecure_channel("localhost:50051") as ch:
        stub = ChatStub(ch)
        responses = []
        async for resp in stub.Chat(client_messages()):
            responses.append(resp)

        assert len(responses) >= 3

Step 6 - Deadline propagation

Per the gRPC core concepts docs (opens in new window), "Clients specify maximum wait time; RPCs terminate with DEADLINE_EXCEEDED if exceeded."

def test_deadline_returns_correct_status():
    with grpc.insecure_channel("localhost:50051") as ch:
        stub = SlowStub(ch)
        with pytest.raises(grpc.RpcError) as exc_info:
            stub.SlowOperation(SlowRequest(), timeout=0.5)

        assert exc_info.value.code() == grpc.StatusCode.DEADLINE_EXCEEDED

Verify the server-side:

def test_server_observes_deadline_propagation():
    # Service should respect deadline and cancel its own downstream calls
    with grpc.insecure_channel("localhost:50051") as ch:
        stub = OrchestratorStub(ch)
        with pytest.raises(grpc.RpcError):
            stub.Compose(ComposeRequest(), timeout=0.1)

    # Verify downstream call observed the cancellation
    downstream_state = fetch_downstream_state()
    assert downstream_state.cancelled_count >= 1

Step 7 - Cancellation behavior

Per the gRPC core concepts docs (opens in new window), "Either party can terminate an RPC immediately. Changes made before a cancellation are not rolled back."

def test_cancellation_is_observed_server_side():
    with grpc.insecure_channel("localhost:50051") as ch:
        stub = LongRunningStub(ch)
        future = stub.LongOperation.future(LongRequest())
        time.sleep(0.5)
        future.cancel()

        # Server should record cancellation
        time.sleep(0.5)
        state = fetch_server_metrics()
        assert state.cancelled_count >= 1

Status codes, metadata, and load testing

See status-codes-metadata-load.md (opens in new window) for the full status-code matrix (OK, CANCELLED, DEADLINE_EXCEEDED, INVALID_ARGUMENT, UNAVAILABLE, ...), a request/response metadata round-trip test, and load testing with ghz.

Worked example

A prices service exposes SubscribePrices, a server-streaming RPC. QA needs to confirm the client receives ordered ticks and that cancelling the stream is observed server-side.

  1. Start with the unary sanity call (Step 2) to confirm the channel and stubs are wired.
  2. Open the stream with a 10s deadline and read 5 ticks: stream = stub.SubscribePrices(SubscribeRequest(symbol="AAPL"), timeout=10.0).
  3. After the 5th tick, call stream.cancel() and break (Step 3).
  4. Assert len(ticks) == 5 and every t.symbol == "AAPL".
  5. Add a cancellation check (Step 7): fetch server metrics and assert cancelled_count >= 1, proving the server observed the client cancel rather than orphaning work.

Result: the ticker stream is verified for ordered delivery, a clean 10s deadline, and server-side cancellation - the three behaviors a server-streaming RPC most often regresses on.

Anti-patterns

Anti-patternWhy it failsFix
Skip deadline + cancellation testsProduction cancellation orphans server-side workSteps 6 + 7
Test only OK and INTERNAL pathsStatus-code regressions go silentlyTest the matrix (status-codes reference)
Use BatchSpanProcessor or similar buffering on test clientStreams "complete" before all messages flushAlways synchronous in tests
Tests share a single channel across goroutinesChannel state contamination flakesPer-test channel
Generate proto stubs at test runtimeCI flakes on plugin churnGenerate in build phase + commit

Limitations

  • gRPC-Web uses HTTP/1.1 fallback; some streaming patterns (client/bidi) are not supported. Test gRPC-Web specifically if used.
  • Long-lived bidi streams hide individual-message error codes; channel-level state matters more.
  • ghz protobuf reflection requires the server to enable reflection service (not always on in production builds).

References

  • gRPC core concepts docs (opens in new window) - RPC patterns, deadlines, cancellation, status codes, metadata
  • websocket-tests (qa-realtime-protocols) - WebSocket alternative for non-gRPC stacks
  • server-sent-events-tests (qa-realtime-protocols) - one-way HTTP streaming alternative

Related skills

buf-cli-lint-breaking-build

Wraps the buf CLI for protobuf PR gating: `buf build` (compile .proto), `buf lint` (STANDARD rules: snake_case fields, Service suffix), `buf breaking --against {ref}` (detect wire/codegen breakage vs a git/BSR baseline), and `buf format`. Use as the CI proto-lint + breaking-change gate, or to debug a breaking failure by rule ID (e.g. FIELD_NO_DELETE_UNLESS_NUMBER_RESERVED) and pick the FILE/PACKAGE/WIRE_JSON/WIRE ruleset per consumer. This is the detection TOOL that enforces the rules and carries the catalog of what is breaking and why (field-number reservation, wire-safe vs wire-incompatible changes, oneof/map constraints, the four buf categories) in references/versioning-strategy.md; for cross-service schema contract testing use protobuf-compat-checking - not this.

ghz-load

Wraps ghz, the gRPC load testing tool, for throughput and latency benchmarking. Covers test invocation (--proto + --call + host:port; or --protoset for compiled descriptors), load parameters (-n total requests, -c concurrency, -r RPS rate limit, -z duration), output formats (json/csv/html/influx-summary for CI consumption), the metrics reported (RPS achieved, latency p50/p95/p99, status-code distribution, errors), and CI integration patterns for regression gating. Use when benchmarking a gRPC service's throughput or detecting latency regressions in CI.

grpc-mock

Wraps gRPC server-mocking patterns for client-side tests: Go bufconn (in-memory net.Listener via google.golang.org/grpc/test/bufconn) + mockgen-generated interface mocks, Python pytest-grpc fixtures + unittest.mock patching of stubs, JVM grpc-mock library / in-process gRPC server (InProcessServerBuilder), Node @grpc/grpc-js fake server with NewServer-on-port-0. Also carries the interceptor-layer test patterns (Go / Java / grpc-js auth, retry, logging, error-mapping, chained ordering via a spy handler) in references/interceptors.md. Use when writing client-side tests that need a controllable gRPC server response (success cases, error cases, timeouts, single-response error injection) without spinning up a real backend, or when testing a gRPC interceptor. For multi-message streaming-sequence tests (server-streaming, bidi), use grpc-streaming-test-author instead. Distinct from grpcurl-cli (ad-hoc CLI invocation against a real server) and ghz-load (perf against a real server).

grpcurl-cli

Wraps grpcurl, the curl-equivalent CLI for gRPC. Covers descriptor sources (server reflection default, --import-path + --proto for proto files, --protoset for compiled descriptor sets), service discovery (`list`, `describe`), invoking unary RPCs (`-d '{...}'`, `-d @file.json`, `-d @` for stdin), streaming RPCs (newline-delimited JSON via stdin), TLS configuration (--cacert, --cert, --key, --insecure, --plaintext), header injection (-H 'Authorization: Bearer ...'), and exit codes. Use for ad-hoc gRPC debugging, smoke testing, scriptable PR-time gates, and CLI-based interaction with reflective gRPC services.