opentelemetry-trace-assertions

Use when a service is instrumented with OpenTelemetry and alerts, SLOs or dashboards depend on span names or attributes a refactor could drop. Captures spans with the in-memory exporter in tests (Python, JS, Java) and asserts on names, attributes, status, kind and structure.

/plugin install qa-distributed-tracing@testland-qa

First time in Claude Code? Add the marketplace with /plugin marketplace add testland/qa

View source (opens in a new tab)

Overview

Tests capture the spans the code under test emits with an in-memory exporter, then assert the contract that alerts, SLOs and dashboards read: span names, attribute keys and values, status, kind, and which span is whose parent. A refactor that drops any of these stays green in ordinary tests and silently empties a panel or stops an alert.

When to use

  • Alerts, SLO queries or dashboards select on span names or attributes.
  • A trace-shape change has already caused a missed page or an empty panel.
  • New instrumentation needs a regression baseline.

Step 1 - Install the SDK test pieces

LanguageInstall
Pythonpip install opentelemetry-sdk
JS / TSnpm install -D @opentelemetry/sdk-trace-base @opentelemetry/sdk-trace-node @opentelemetry/api
Javaio.opentelemetry:opentelemetry-sdk-testing (test scope)
.NETdotnet add package OpenTelemetry.Exporter.InMemory

Step 2 - Wire a synchronous in-memory exporter

SimpleSpanProcessor passes each span to the exporter as it ends; the batching processor exports on a timer, so a read right after the exercise misses spans and tests start sleeping (Python SDK export docs (opens in new window)).

# tests/conftest.py
import pytest
from opentelemetry import trace
from opentelemetry.sdk.trace import TracerProvider
from opentelemetry.sdk.trace.export import SimpleSpanProcessor
from opentelemetry.sdk.trace.export.in_memory_span_exporter import InMemorySpanExporter

_exporter = InMemorySpanExporter()
_provider = TracerProvider()
_provider.add_span_processor(SimpleSpanProcessor(_exporter))
trace.set_tracer_provider(_provider)        # once per process

@pytest.fixture
def spans():
    _exporter.clear()                        # each test sees only its own spans
    def by_name(name):
        found = [s for s in _exporter.get_finished_spans() if s.name == name]
        assert found, f"no finished span named {name!r}"
        return found[-1]
    yield by_name
    _exporter.clear()

Reach spans through by_name, never by index or by counting them: instrumentation order and extra spans change without the contract changing, and an empty buffer makes loops over spans pass vacuously.

Step 3 - Assert name, attributes, status and kind

from opentelemetry.trace import SpanKind, StatusCode

def test_order_create_span(spans):
    create_order(items=[item])

    order = spans("order.create")
    assert order.attributes["order.item_count"] == 1
    assert order.status.status_code != StatusCode.ERROR
    assert order.kind == SpanKind.INTERNAL

def test_rates_provider_down_is_an_error(spans, provider_down):
    price = convert(100, "EUR")              # falls back to the cached rate
    assert price == 8600                     # the conversion still succeeds

    fetch = spans("rates.fetch")
    assert fetch.kind == SpanKind.CLIENT     # an outgoing call, not in-process
    assert fetch.status.status_code == StatusCode.ERROR
    assert fetch.attributes["error.type"] == "ConnectionRefusedError"
    assert spans("pricing.convert").status.status_code != StatusCode.ERROR

Status rules for HTTP spans: leave status unset on 1xx-3xx and on server-side 4xx; set Error on client-side 4xx and on 5xx (HTTP semantic conventions (opens in new window)). error.type is required when the call ended in an error. Only the failing span gets the error status; a caller that recovered keeps a successful one.

Step 4 - Assert parent-child structure

IDs are random per run, so compare IDs read from the same run, never literal values:

def test_db_span_is_child_of_order(spans):
    create_order(items=[item])
    order, db = spans("order.create"), spans("db.query")

    assert db.context.trace_id == order.context.trace_id
    assert db.parent is not None, "db.query started a new trace"
    assert db.parent.span_id == order.context.span_id

For context sent to another service, parse the outgoing traceparent header the fake client captured (00-<trace_id>-<span_id>-<flags>) and compare its trace and span IDs with the CLIENT span's own context.trace_id / context.span_id formatted as 32 and 16 hex digits (W3C Trace Context (opens in new window)). A span for another service's work (inventory, a worker) cannot be proved here; do not create one in the test and assert on it.

Step 5 - Semantic-convention keys

HTTP client spans require http.request.method, url.full, server.address and server.port; http.response.status_code is required when a status was received and error.type when the call failed. The span name is {method}, or {method} {target} when a low-cardinality target is known (HTTP semantic conventions (opens in new window)):

def test_charge_call_attributes(spans):
    charge_card(order)
    http = spans("POST")                     # "{method}" or "{method} {url.template}"
    assert http.kind == SpanKind.CLIENT
    assert http.attributes["http.request.method"] == "POST"
    assert http.attributes["url.full"].startswith("https://gateway.example.com/charges")
    assert http.attributes["server.address"] == "gateway.example.com"
    assert http.attributes["http.response.status_code"] == 201

Older instrumentation emits the experimental keys (http.method, http.status_code). OTEL_SEMCONV_STABILITY_OPT_IN=http switches to the stable keys and http/dup emits both during a migration (HTTP semantic conventions (opens in new window)). Before switching, find every alert and dashboard that selects on the old keys: they stop matching the day the change ships.

Step 6 - JS / TS

JS SDK 2.x removed addSpanProcessor(); pass processors to the provider constructor (upgrade guide (opens in new window)).

import { InMemorySpanExporter, SimpleSpanProcessor } from '@opentelemetry/sdk-trace-base';
import { NodeTracerProvider } from '@opentelemetry/sdk-trace-node';
import { SpanKind, SpanStatusCode } from '@opentelemetry/api';

const exporter = new InMemorySpanExporter();
new NodeTracerProvider({ spanProcessors: [new SimpleSpanProcessor(exporter)] }).register();

const byName = (name: string) => {
  const span = exporter.getFinishedSpans().findLast((s) => s.name === name);
  if (!span) throw new Error(`no finished span named ${name}`);
  return span;
};

beforeEach(() => exporter.reset());

test('rates.fetch records the outage', async () => {
  await convert(100, 'EUR');
  const fetch = byName('rates.fetch');
  expect(fetch.kind).toBe(SpanKind.CLIENT);
  expect(fetch.status.code).toBe(SpanStatusCode.ERROR);
  expect(fetch.attributes['error.type']).toBe('ECONNREFUSED');
});

Step 7 - Java (JUnit 5)

@RegisterExtension
static final OpenTelemetryExtension otel = OpenTelemetryExtension.create();

@Test
void orderCreateEmitsSpan() {
    new OrderService(otel.getOpenTelemetry()).create(order);

    SpanData span = otel.getSpans().stream()
        .filter(s -> s.getName().equals("order.create"))
        .findFirst().orElseThrow();
    assertThat(span.getAttributes().get(AttributeKey.longKey("order.item_count"))).isEqualTo(1L);
}

The extension resets captured spans before each test; inject its OpenTelemetry into the code under test (Java SDK testing (opens in new window)).

Step 8 - CI

Pin the SDK and instrumentation versions so attribute keys do not change between runs, and run trace tests in the normal test job: with the synchronous processor they add milliseconds, not a separate slow job.

- run: pip install -r requirements-test.txt   # opentelemetry-sdk pinned there
- run: pytest tests -v

Anti-patterns

Anti-patternWhy it failsFix
BatchSpanProcessor in testsSpans missing at read time; tests sleepSimpleSpanProcessor (Step 2)
No reset between testsOne test reads another's spansClear in the fixture (Step 2)
spans[0], len(spans) == 3Breaks on harmless changes; hides missing spansLook up by name (Step 2)
Asserts inside for s in spans:Passes when nothing was capturedResolve the span unconditionally
Literal trace / span IDsRandom per runCompare IDs from the same run (Step 4)
Remote call left as INTERNALRules selecting CLIENT / PRODUCER miss itAssert kind (Step 3)
Catch-and-fallback leaves the failing span unsetAvailability SLO counts the outage as successError status + error.type on that span (Step 3)
Editing the alert or dashboard to match what the code emitsHides the regressionFix the instrumentation

Limitations

  • In-process only: it cannot prove spans emitted by another service. Use jaeger-trace-tests against a real backend for cross-service traces.
  • Span links and events are not covered by these patterns.
  • Instrumentation library upgrades can change keys; pin versions and re-run on upgrade.

References

  • Six hours of stale exchange rates, and the availability panel never moved
  • Northbank was down for 41 minutes and our page never fired
  • A month of an empty database panel, and the trace tests never went red
  • Closing OBS-3140 before Friday's freeze
  • Our two trace test files cost more wall clock than the whole rest of CI