Jaeger query API, advanced assertions, and CI wiring

Deep reference for the jaeger-trace-tests SKILL.md. Consult for the full Jaeger query API surface, parent-child + duration assertion patterns, the GitHub Actions service block, and per-test isolation / retention on a shared CI backend.

Query API endpoints

Jaeger exposes trace data over an HTTP query API on :16686:

Trace JSON response shape (selected fields):

{
  "data": [{
    "traceID": "abc...",
    "spans": [
      {
        "spanID": "def...",
        "operationName": "order.create",
        "duration": 12345,
        "tags": [{"key": "order.item_count", "type": "int64", "value": 1}],
        "references": [{"refType": "CHILD_OF", "spanID": "parent..."}]
      }
    ]
  }]
}

duration is in microseconds; tags is a flat list of typed key/value pairs; parent links live in references, not on the child span directly.

Parent-child assertions via references

Jaeger encodes parent links as references with refType: "CHILD_OF".

def parent_id(span):
    refs = span.get("references", [])
    child_of = [r for r in refs if r["refType"] == "CHILD_OF"]
    return child_of[0]["spanID"] if child_of else None

assert parent_id(db_span) == order_span["spanID"]

Assert a duration ceiling straight off the microsecond duration field:

assert order_span["duration"] < 500_000  # under 500ms

GitHub Actions service

Run the all-in-one image as a job service so every step can reach OTLP ingest and the query API:

services:
  jaeger:
    image: cr.jaegertracing.io/jaegertracing/jaeger:2.17.0
    ports:
      - 16686:16686
      - 4317:4317
      - 4318:4318

Full port map: 16686 query API + UI, 4317 OTLP/gRPC ingest, 4318 OTLP/HTTP ingest, 5778 sampling config, 9411 Zipkin (B3) compatibility.

Per-test isolation on shared CI

CI runs many tests against one Jaeger. Scope each test with a unique service.name (or a unique trace tag) so a query never sees another test's spans:

service_name = f"orders-test-{uuid4()}"
# configure the SDK with this service name, then query filtered by it

In-memory storage is bounded by Jaeger's eviction; long test runs should restart the container or accept eviction.

Retention

All-in-one uses transient in-memory storage per the Jaeger getting-started docs (opens in new window). For longer runs mount a config or restart the container between workflows:

docker run ... \
  -v /path/to/config.yaml:/jaeger/config.yaml \
  cr.jaegertracing.io/jaegertracing/jaeger:2.17.0 \
  --config /jaeger/config.yaml

Production storage backends (Cassandra, Elasticsearch, OpenSearch, Badger) matter for real deployments; the in-memory all-in-one is sufficient for CI.

Grafana Tempo backend - TraceQL queries, same workflow

Grafana Tempo ("an open source and high-scale distributed tracing backend", per Tempo getting started (opens in new window)) follows the same run-query-assert workflow, with two differences: queries use TraceQL, and structural operators give parent-child / descendant assertions Jaeger's flat span-list API does not expose natively.

Run single-binary in CI

Ports: 3200 (HTTP API + UI), 4317/4318 (OTLP ingest). Tempo needs a minimal tempo.yaml (per the configuration reference (opens in new window), monolithic target: all):

# tempo.yaml
server:
  http_listen_port: 3200
distributor:
  receivers:
    otlp:
      protocols:
        grpc: { endpoint: 0.0.0.0:4317 }
        http: { endpoint: 0.0.0.0:4318 }
storage:
  trace:
    backend: local
    local: { path: /var/tempo/traces }

docker run --rm --name tempo -p 3200:3200 -p 4317:4317 -p 4318:4318 \
  -v "$PWD/tempo.yaml:/etc/tempo.yaml" \
  grafana/tempo:latest -target=all -config.file=/etc/tempo.yaml

until curl -sf http://localhost:3200/ready; do sleep 1; done   # readiness gate

The SDK exporter config is identical to Jaeger's (OTLP to :4317).

TraceQL essentials

Per Construct a TraceQL query (opens in new window), span selectors use { }:

Operators: =, !=, >, >=, <, <=, =~ (regex, fully anchored - wrap with .* for partial match), !~; connectives &&, ||.

Structural operators assert span relationships: >> descendant, > direct child, << ancestor, ~ sibling:

{ span.http.url = "/checkout" } >> { span.db.system = "postgresql" }

Pipelines aggregate: { span:status = error } | count() > 1, { resource.service.name = "api" } | avg(span:duration) > 500ms.

Query via /api/search

Per the Tempo API docs (opens in new window), /api/search takes q (URL-encoded TraceQL), limit (default 20), start/end (epoch seconds), spss (spans per span-set, default 3), minDuration/maxDuration.

def test_checkout_span_reaches_tempo():
    with tracer.start_as_current_span("POST /order"):
        place_order(items=["widget"])
    trace.get_tracer_provider().force_flush(timeout_millis=5000)
    time.sleep(0.5)

    query = '{ resource.service.name = "checkout" && span.http.url = "/order" }'
    traces = requests.get("http://localhost:3200/api/search",
                          params={"q": query, "limit": 1}).json()["traces"]
    assert len(traces) == 1
    assert traces[0]["rootServiceName"] == "checkout"

Full trace by ID - parent-child assertions

GET /api/traces/{traceID} returns OpenTelemetry JSON; walk resourceSpans[].scopeSpans[].spans[] and assert db_span["parentSpanId"] == root_span["spanId"]. Attributes are { "key": ..., "value": { "<type>Value": ... } } objects (OTel proto format).

Tempo-specific anti-patterns

Limitations

• backend: local suits CI, not production (object storage recommended per the configuration reference).
• TraceQL requires the Parquet block format (Tempo's default); legacy TSDB blocks don't support it.

References

• Tempo getting started (opens in new window)
• TraceQL overview (opens in new window)
• Construct a TraceQL query (opens in new window)
• Tempo HTTP API docs (opens in new window)
• Tempo configuration reference (opens in new window)
• Single-binary docker-compose example (opens in new window)

Zipkin backend - same run-query-assert workflow

Zipkin is the original distributed-tracing system (predates OpenTelemetry); still common in Java shops via Spring Cloud Sleuth heritage, and these tests protect a Zipkin → Jaeger/OTel cutover. The workflow is identical to Jaeger's: run the backend in CI, ship spans, force_flush(), query the REST API, assert on the span set.

Run in CI

Per the Zipkin quickstart (opens in new window):

docker run -d -p 9411:9411 openzipkin/zipkin

services:
  zipkin:
    image: openzipkin/zipkin
    ports: ["9411:9411"]

REST API

Per the Zipkin API spec (opens in new window):

Ship + query + assert

from opentelemetry.exporter.zipkin.json import ZipkinExporter
# BatchSpanProcessor(ZipkinExporter(endpoint="http://localhost:9411/api/v2/spans"))

def test_order_trace_in_zipkin():
    with use_tracer():
        create_order(items=[item])
    trace.get_tracer_provider().force_flush(timeout_millis=5000)
    time.sleep(0.5)

    traces = requests.get(
        "http://localhost:9411/api/v2/traces",
        params={"serviceName": "orders", "spanName": "order.create",
                "lookback": 60000, "limit": 1},
    ).json()                       # list of lists of spans
    assert len(traces) == 1
    span = next(s for s in traces[0] if s["name"] == "order.create")
    assert span["tags"]["order.item_count"] == "1"   # Zipkin V2 tags are ALL strings

Zipkin V2 stores tag values as strings (vs Jaeger's typed tags) - cast in assertions accordingly.

B3 propagation header tests

Per the B3 propagation spec (opens in new window):

• Multi-header: X-B3-TraceId (32/16 lower-hex), X-B3-SpanId (16), X-B3-ParentSpanId (absent on root), X-B3-Sampled (1/0), X-B3-Flags (1 debug).
• Single-header: b3: {TraceId}-{SpanId}-{SamplingState}-{ParentSpanId}, sampling 1 accept / 0 deny / d debug / absent defer.

Test both forms - modern services increasingly send only the single-header form:

def test_b3_single_header_propagates():
    headers = {"b3": f"{trace_id}-{span_id}-1-{parent_id}"}
    requests.get("http://localhost:8080/orders", headers=headers)
    time.sleep(0.5)
    spans = requests.get(f"http://localhost:9411/api/v2/trace/{trace_id}").json()
    assert any(s["traceId"] == trace_id for s in spans)

Dependency-graph assertion

Zipkin computes service dependencies from observed traces; aggregation is lazy (in-memory computes inline; Cassandra uses Spark batch) - allow

=2s:

deps = requests.get("http://localhost:9411/api/v2/dependencies",
                    params={"endTs": int(time.time() * 1000), "lookback": 60000}).json()
pair = next((d for d in deps if d["parent"] == "orders" and d["child"] == "payments"), None)
assert pair and pair["callCount"] >= 1

Zipkin-specific anti-patterns

References

• Zipkin quickstart (opens in new window)
• Zipkin API spec (opens in new window)
• B3 propagation spec (opens in new window)