tempo-trace-tests

Authors integration tests that query Grafana Tempo for cross-service trace verification - TraceQL `{ }` span selectors targeting `span.`, `resource.`, and intrinsic fields; Tempo HTTP API (`/api/search` with `q=`, `/api/traces/{id}`) for span-set and attribute assertions; local Tempo via Docker single-binary (ports 4317/4318/3200). Use when the production observability stack uses Tempo as the trace backend and tests must verify distributed trace shape, span attributes, or service topology after instrumentation changes.

tempo-trace-tests

Grafana Tempo is "an open source and high-scale distributed tracing backend" (Grafana Tempo getting started). Its native query language, TraceQL, is "designed for selecting traces in Tempo" (TraceQL overview). This skill covers authoring tests that run TraceQL queries and HTTP API calls against a local Tempo instance to assert on span attributes, service topology, and trace duration.

When to use

E2E or integration test exercises multiple services and you need to verify the distributed trace shape (not just per-process spans).
Production stack uses Tempo; tests should exercise the same TraceQL expressions your dashboards and alerts depend on.
Smoke test after instrumentation changes - confirm spans reach Tempo and carry the correct attributes.
You need structural assertions (parent-child, descendant) that Jaeger's flat span list API does not expose natively.

Step 1 - Run Tempo single-binary in CI

The grafana/tempo single-binary example exposes (docker-compose/single-binary):

Port	Purpose
3200	Tempo HTTP API + UI
4317	OTLP/gRPC ingest
4318	OTLP/HTTP ingest

Tempo requires a minimal tempo.yaml. The configuration reference (Tempo configuration) specifies these required sections for monolithic mode - Kafka is not needed when target: all:

# tempo.yaml
server:
  http_listen_port: 3200

distributor:
  receivers:
    otlp:
      protocols:
        grpc:
          endpoint: 0.0.0.0:4317
        http:
          endpoint: 0.0.0.0:4318

storage:
  trace:
    backend: local
    local:
      path: /var/tempo/traces

Docker run (bind-mount the config):

docker run --rm --name tempo \
  -p 3200:3200 \
  -p 4317:4317 \
  -p 4318:4318 \
  -v "$PWD/tempo.yaml:/etc/tempo.yaml" \
  grafana/tempo:latest \
  -target=all -config.file=/etc/tempo.yaml

GitHub Actions service (docker-compose style):

services:
  tempo:
    image: grafana/tempo:latest
    command: ["-target=all", "-config.file=/etc/tempo.yaml"]
    ports:
      - "3200:3200"
      - "4317:4317"
      - "4318:4318"
    volumes:
      - ./tempo.yaml:/etc/tempo.yaml

Wait for readiness before running tests. The /ready endpoint (Tempo API docs) returns HTTP 200 when Tempo is ready to serve traffic:

until curl -sf http://localhost:3200/ready; do sleep 1; done

Step 2 - Configure the SDK to export to Tempo

from opentelemetry import trace
from opentelemetry.sdk.trace import TracerProvider
from opentelemetry.sdk.trace.export import BatchSpanProcessor
from opentelemetry.exporter.otlp.proto.grpc.trace_exporter import OTLPSpanExporter

provider = TracerProvider()
provider.add_span_processor(
    BatchSpanProcessor(
        OTLPSpanExporter(endpoint="http://localhost:4317", insecure=True)
    )
)
trace.set_tracer_provider(provider)

Use force_flush() before querying - Step 4 covers the flush pattern.

Step 3 - TraceQL query syntax

TraceQL span selectors use { } braces. The simplest selector { } matches all spans. Conditions inside the braces filter spans (Construct a TraceQL query).

Attribute scopes

Prefix	Meaning	Example
`span.`	Span-scoped attribute	`span.http.status_code`
`resource.`	Resource attribute	`resource.service.name`
`span:`	Intrinsic span field	`span:status`, `span:duration`, `span:name`, `span:kind`
`trace:`	Trace-level intrinsic	`trace:duration`, `trace:rootService`, `trace:rootName`

Comparison operators

=, !=, >, >=, <, <=, =~ (regex, fully anchored), !~ (negated regex).

{ span.http.status_code >= 400 && span.http.status_code < 500 }
{ resource.service.name = "checkout" && span:status = error }
{ span.http.method =~ "GET|POST" }
{ span.any_attribute != nil }
{ trace:duration > 2s }

Logical connectives within a selector: && (AND), || (OR).

Structural operators

These assert on span relationships within a trace (Construct a TraceQL query):

Operator	Meaning
`>>`	Descendant (any depth)
`>`	Direct child
`<<`	Ancestor
`~`	Sibling

{ span.http.url = "/checkout" } >> { span.db.system = "postgresql" }

This selects traces where a checkout HTTP span has a PostgreSQL descendant span.

Pipeline operators

Chain aggregations with | (Construct a TraceQL query):

{ span:status = error } | count() > 1
{ span:status = error } | by(resource.service.name) | count() > 1
{ resource.service.name = "api" } | avg(span:duration) > 500ms
{ span:status = error } | select(span.http.status_code, span.http.url)

Step 4 - Query via `/api/search`

The /api/search endpoint accepts a q parameter containing a URL-encoded TraceQL expression. Per the Tempo API docs:

Parameter	Default	Notes
`q`	-	URL-encoded TraceQL query
`limit`	20	Maximum traces returned
`start` / `end`	-	Unix epoch seconds; scopes search to a time range
`spss`	3	Spans per span-set in the response
`minDuration` / `maxDuration`	-	Go duration format (e.g. `100ms`, `2s`)

Response shape (abridged):

{
  "traces": [
    {
      "traceID": "abc123",
      "rootServiceName": "checkout",
      "rootTraceName": "POST /order",
      "durationMs": 342,
      "spanSets": [
        {
          "spans": [
            {
              "spanID": "def456",
              "durationNanos": "45000000",
              "attributes": [
                { "key": "http.status_code", "value": { "intValue": "200" } }
              ]
            }
          ],
          "matched": 1
        }
      ]
    }
  ]
}

Force-flush + brief sleep before querying so BatchSpanProcessor ships all pending spans:

import time, requests
from urllib.parse import quote

def test_checkout_span_reaches_tempo():
    with tracer.start_as_current_span("POST /order"):
        place_order(items=["widget"])

    trace.get_tracer_provider().force_flush(timeout_millis=5000)
    time.sleep(0.5)  # Tempo ingest pipeline

    query = '{ resource.service.name = "checkout" && span.http.url = "/order" }'
    resp = requests.get(
        "http://localhost:3200/api/search",
        params={"q": query, "limit": 1},
    )
    resp.raise_for_status()
    traces = resp.json()["traces"]
    assert len(traces) == 1, f"Expected 1 trace, got {len(traces)}"
    assert traces[0]["rootServiceName"] == "checkout"

Step 5 - Fetch a full trace by ID via `/api/traces/{id}`

GET /api/traces/{traceID} returns the full trace in OpenTelemetry JSON format (proto spec at opentelemetry-proto) (Tempo API docs). Optional start/end epoch-second params scope the backend search window.

def test_db_span_is_child_of_checkout():
    with tracer.start_as_current_span("POST /order") as root:
        with tracer.start_as_current_span("db.query"):
            run_query("INSERT INTO orders ...")
        trace_id = format(root.get_span_context().trace_id, "032x")

    trace.get_tracer_provider().force_flush(timeout_millis=5000)
    time.sleep(0.5)

    resp = requests.get(f"http://localhost:3200/api/traces/{trace_id}")
    resp.raise_for_status()
    data = resp.json()

    resource_spans = data["resourceSpans"]
    all_spans = [
        s
        for rs in resource_spans
        for ss in rs["scopeSpans"]
        for s in ss["spans"]
    ]

    root_span = next(s for s in all_spans if s["name"] == "POST /order")
    db_span   = next(s for s in all_spans if s["name"] == "db.query")

    assert db_span["parentSpanId"] == root_span["spanId"]

Step 6 - Assert span attributes

Attributes in the OTel JSON live in each span's attributes array as { "key": "...", "value": { "<type>Value": ... } } objects (OTel proto format, per Tempo API docs).

def attr(span, key):
    for a in span.get("attributes", []):
        if a["key"] == key:
            v = a["value"]
            return next(iter(v.values()))
    return None

assert attr(db_span, "db.system") == "postgresql"
assert int(attr(root_span, "http.status_code")) == 200

Step 7 - Per-test isolation

CI runs many tests against one Tempo instance. Use a unique resource.service.name per test run to scope /api/search queries:

import uuid
service = f"checkout-test-{uuid.uuid4()}"
# ... configure TracerProvider with service.name = service ...
query = f'{{ resource.service.name = "{service}" }}'

Tempo local storage is in-memory-like; long test runs should restart the container or use start/end time bounds in search calls.

Anti-patterns

Anti-pattern	Why it fails	Fix
Query `/api/search` before flush	`BatchSpanProcessor` defers export; spans not yet ingested	`force_flush()` + brief sleep (Step 4)
Omit `start`/`end` on long CI runs	Searches all backend blocks; slow and noisy	Pass epoch bounds scoped to the test window
Use production Tempo from CI	Test traces pollute real data; resource constraints	Always use Docker single-binary (Step 1)
Hard-code service name across tests	Cross-test contamination on shared instance	Unique `service.name` per test run (Step 7)
Assert span count from `/api/search` with default `spss=3`	`spss` caps spans per span-set; count is not total span count	Fetch full trace via `/api/traces/{id}` for complete span set
Use Grafana UI as the assertion surface	Scraping HTML is fragile; UI is for humans	Use `/api/search` and `/api/traces/{id}` in tests

Limitations

Tempo local storage (backend: local) is suitable for CI but not production. The configuration reference recommends object storage for production workloads (Tempo configuration).
TraceQL requires the Parquet columnar block format, which is Tempo's default (TraceQL overview). Legacy TSDB block format does not support TraceQL.
/api/search returns at most limit traces (default 20) and spss spans per span-set (default 3). For complete span enumeration use /api/traces/{id} (Tempo API docs).
Regex in TraceQL (=~) is fully anchored - partial matching requires wrapping: =~ ".*substring.*" (Construct a TraceQL query).

References

Grafana Tempo getting started - overview and deployment modes
TraceQL overview - language purpose and Parquet requirement
Construct a TraceQL query - selectors, operators, structural operators, pipelines
Tempo HTTP API docs - /api/search, /api/traces/{id}, /ready, /api/echo endpoints
Tempo configuration reference - server, distributor, storage sections
Tempo docker-compose single-binary example - image, ports, volume mounts
jaeger-trace-tests - sister skill for Jaeger-using teams
zipkin-trace-tests - sister skill for Zipkin-using teams
opentelemetry-trace-assertions - in-process unit-level trace assertions
trace-coverage-reviewer - adversarial reviewer for trace coverage gaps

tempo-trace-tests

tempo-trace-tests

When to use

Step 1 - Run Tempo single-binary in CI

Step 2 - Configure the SDK to export to Tempo

Step 3 - TraceQL query syntax

Attribute scopes

Comparison operators

Structural operators

Pipeline operators

Step 4 - Query via /api/search

Step 5 - Fetch a full trace by ID via /api/traces/{id}

Step 6 - Assert span attributes

Step 7 - Per-test isolation

Anti-patterns

Limitations

References

Step 4 - Query via `/api/search`

Step 5 - Fetch a full trace by ID via `/api/traces/{id}`