langfuse-tracing
Wires Langfuse tracing into LLM apps for production observability, monitoring, telemetry, and offline eval - instruments via `@observe` (Python) / `startActiveObservation` (TS) decorators that auto-capture inputs / outputs / timings / errors per generation; exposes `langfuse.update_current_span()` for metadata + cost / latency annotation; supports trace-bound scoring for eval datasets and prompt-as-code management. Use when the user needs to monitor, log, trace, or debug LLM API calls in production beyond pre-deploy eval, wants to add LLM observability tooling to an existing app, or wants to ship traces from production to an eval dataset for offline regression testing.
Install with skills.sh (any agent)
npx skills add testland/qa --skill langfuse-tracinglangfuse-tracing
Overview
Langfuse is a production LLM observability platform (per lf-gh (opens in new window)).
Step 1 - Install
Per lf-gh (opens in new window):
pip install langfuseFor TypeScript:
npm install @langfuse/tracingSet up project credentials per Langfuse self-hosted or cloud project (LANGFUSE_PUBLIC_KEY + LANGFUSE_SECRET_KEY + LANGFUSE_HOST).
Step 2 - Instrument with @observe
Per langfuse.com/docs/sdk/python/decorators (opens in new window):
Python:
from langfuse import observe
@observe(name="llm-call", as_type="generation") # auto-captures inputs, outputs, timings, errors
async def my_async_llm_call(prompt_text):
return "LLM response"TypeScript - create an observation, do work inside it, then end it:
import { startActiveObservation } from "@langfuse/tracing";
const { observation, end } = startActiveObservation({
name: "llm-call",
type: "generation",
input: { prompt: promptText },
});
try {
const result = await callMyLLM(promptText);
observation.update({ output: result });
return result;
} finally {
end();
}Validation: After your first instrumented call, open the Langfuse UI → Traces. You should see a new trace with nested observations, inputs, outputs, and timings. If no trace appears within ~30 s:
Step 3 - Update current observation with metadata
Per lf-py-deco (opens in new window):
from langfuse import get_client
langfuse = get_client()
with langfuse.start_as_current_observation(as_type="span", name="data-processing"):
langfuse.update_current_span(metadata={"step1_complete": True})Common metadata fields used in production:
Step 4 - Score traces
Per langfuse.com/docs/scores (opens in new window):
langfuse.score(
trace_id="...",
name="answer_relevance",
value=0.87, # numeric (0-1); also supports categorical (string) and boolean
comment="Judged by GPT-4 rubric"
)Scores can come from:
Validation: After scoring, open the Langfuse UI → Traces → select the trace. The score should appear in the Scores panel. If missing, confirm the trace_id matches an existing trace and that the score call did not raise an exception.
Step 5 - Datasets for offline eval
Datasets are collections of (input, expected_output) items built from production traces, CSV / JSONL imports, or the UI. Run one against your app to diff vs a baseline:
items = langfuse.get_dataset_items(dataset_id="...")
for item in items:
actual = my_llm_app(item.input)
item.run(actual) # links the run back to the dataset for diff vs baselineSee references/datasets.md for building datasets from production traces, the lf-ds (opens in new window) API signature, and the run-validation checkpoint.
Step 6 - Prompt management
Fetch the current production prompt at runtime and pin its version so prompt drift is traceable:
from langfuse import get_client
langfuse = get_client()
# Fetch the production-labelled prompt; falls back to cache if offline
prompt = langfuse.get_prompt("my-prompt-name", label="production")
compiled = prompt.compile(user_input=user_query)
# Pass compiled text to your LLM call
response = my_llm_call(compiled)Iterate prompt text in the Langfuse UI and roll out new versions per environment (production / staging labels) without code deploys. Version history and A/B comparison are available in the UI.
Validation: After an instrumented call using a managed prompt, open the Langfuse UI → Traces → select the trace. The prompt name and version should appear in the trace metadata, confirming prompt version is pinned and attributable.
Step 7 - CI integration
Langfuse is observability-side, not pre-deploy CI-side. Post-deploy CI patterns query the Langfuse API for recent traces and assert on aggregate metrics (a score-query gate, eval-on-trace, cost-regression, score-based alerting). Runnable gate script and the full pattern list: references/ci-integration.md.
Anti-patterns
| Anti-pattern | Why it fails | Fix |
|---|---|---|
| Trace everything in production with no sampling | Cost explodes at scale | Use level=DEBUG + UI-side sampling (Step 3) |
| Score traces only via UI (no automated path) | Can't catch silent regressions | Automated langfuse.score() per trace (Step 4) |
| Pull production trace inputs without privacy review | PII leakage into eval datasets | Cross-ref synthetic-pii-generator for fixture sanitization before promotion |
| Skip prompt versioning | Prompt drift breaks attribution | langfuse.get_prompt() with version pin (Step 6) |
| Conflate Langfuse with pre-deploy eval | Tries to be both; wins neither | Pair Langfuse (post-deploy) with Promptfoo/DeepEval/Ragas (pre-deploy) |
Version notes
Per lf-gh (opens in new window), the SDK was rewritten in v4 and released in March 2026; this skill targets the v4 API. v3 patterns are no longer supported - pin the SDK version in requirements, and for v3 codebases see the upstream migration guide.
Limitations
References
Langfuse CI integration
View source (opens in new window)Langfuse CI integration
Langfuse is observability-side, not pre-deploy CI-side. Typical post-deploy CI patterns use the Langfuse API to query recent traces and assert on aggregate metrics.
Score-query gate
import httpx, os, sys
from datetime import datetime, timedelta, timezone
LANGFUSE_HOST = os.environ["LANGFUSE_HOST"]
headers = {"Authorization": f"Bearer {os.environ['LANGFUSE_SECRET_KEY']}"}
# Fetch average answer_relevance score over the last hour
one_hour_ago = (datetime.now(timezone.utc) - timedelta(hours=1)).isoformat()
resp = httpx.get(
f"{LANGFUSE_HOST}/api/public/scores",
params={"name": "answer_relevance", "fromTimestamp": one_hour_ago},
headers=headers,
)
scores = resp.json()["data"]
avg = sum(s["value"] for s in scores) / len(scores) if scores else 1.0
if avg < 0.75:
print(f"answer_relevance regression: {avg:.2f} < 0.75 threshold")
sys.exit(1)Validation: After the CI job runs, confirm the script exits 0 and that the queried score window covers the expected deployment window. If scores is empty, verify the fromTimestamp range and that instrumented calls have been scored in that period.
Other CI wiring patterns
Langfuse datasets for offline eval
View source (opens in new window)Langfuse datasets for offline eval
Langfuse datasets are collections of (input, expected_output) items. Build them from three sources:
Shipping traces from production into a dataset, then replaying them, is the core offline-regression workflow: item.run() links each replay back to the dataset so the Langfuse UI can diff the run against a baseline. See langfuse.com/docs/datasets (opens in new window) for the current API signature.
Validation
After running a dataset, open the Langfuse UI → Datasets → select your dataset. Each item run should appear under the Runs tab linked to its trace. If runs are missing, confirm dataset_id is correct and that item.run() did not raise an exception.
Related skills
deepeval-evaluation
Authors and runs DeepEval - pytest-native LLM eval framework with `LLMTestCase` (input + actual_output + expected_output + retrieval_context) and ~11 built-in metrics (G-Eval, Answer-Relevancy, Faithfulness, Contextual-Recall / Precision / Relevancy, Hallucination, Bias, Toxicity, Summarization, JSON-Correctness); runs via `deepeval test run {file.py}` with `assert_test()` per test or `evaluate()` for batch; integrates Confident-AI dashboard. Use when the user prefers pytest workflow, works with RAG and needs faithfulness/contextual metrics out-of-the-box, or wants a managed dashboard.
giskard-llm
Authors and runs Giskard LLM scans - adversarial test-case generation for LLM applications via `giskard.scan(model)` covering 7 vulnerability categories (hallucination, harmful_content, prompt_injection, sensitive_information_disclosure, stereotypes, robustness, basic_sycophancy); wraps any callable model behind `giskard.Model(model_predict, model_type="text_generation", ...)`; emits HTML report. Use when the user needs adversarial / red-team coverage on top of functional eval suites.
llm-eval-anti-patterns
Audits an existing LLM evaluation suite for eight methodology errors that make its numbers untrustworthy: too few cases per capability, single-provider lock-in, exact-match assertions on open-ended output, no semantic-similarity check on paraphrase-tolerant output, no baseline comparison in CI, no cost or latency ceiling, unpinned model identifiers, and no adversarial coverage. Supplies harness-neutral detection cues, the reason each error invalidates the result, a concrete fix, a Critical/Warning/Info severity scheme, and a findings-table output shape. Covers validating an LLM judge against human labels before its verdicts count as evidence. Use when an eval suite already exists and its pass rate is about to gate a release, a model swap, or a prompt change, and nobody has audited how the suite itself was built.
llm-regression-suite-author
Builds a versioned golden-dataset LLM regression suite for tracking quality across model upgrades: structures a versioned JSONL/CSV golden dataset, configures deterministic eval runs (temperature 0, seed), wires assertion layers (exact, semantic similarity, LLM-as-judge, rubric), enforces a pass-rate threshold with diff reporting vs the baseline model, and gates CI on regression. Use when upgrading an LLM provider model and needing a repeatable before/after quality gate, or when a prompt regression suite must track output quality across model versions over time.
openai-evals
Authors and runs OpenAI Evals - Python framework + registry for evaluating LLMs and LLM-backed systems with `oaieval {model} {eval-name}` CLI; supports template-based evals (Match / Includes / FuzzyMatch / ModelBasedClassify) defined in `evals/registry/evals/*.yaml` against JSONL data files in `evals/registry/data/`, plus custom Python eval classes implementing the Eval interface. Use when the user works with the openai/evals repo, needs the OpenAI-curated eval registry, or contributes new evals via PR to the registry.
promptfoo-evaluation
Authors and runs Promptfoo evals for LLM prompts and RAG pipelines - wires `promptfooconfig.yaml` providers + prompts + tests + assertions (deterministic `equals` / `contains` / `is-json` / `regex`, semantic `similar`, model-graded `llm-rubric` / `factuality` / `g-eval`, performance `latency` / `cost`, custom `javascript` / `python`), runs `npx promptfoo eval`, views HTML report via `promptfoo view`, and integrates CI for regression gating. Use when the user runs Promptfoo, asks about prompt regression suites, or needs an eval-driven workflow for LLM-backed features.
ragas-evaluation
Authors and runs Ragas - RAG-pipeline evaluation framework with metrics organized into RAG (Faithfulness, Response Relevancy, Context Precision/Recall, Context Entities Recall, Noise Sensitivity), Natural Language Comparison (Factual Correctness, Semantic Similarity, BLEU/ROUGE/CHRF/Exact Match), Agents/Tool-Use (Topic Adherence, Tool Call Accuracy/F1, Agent Goal Accuracy), General Purpose (Aspect Critic, Rubrics-based Scoring), Nvidia (Answer Accuracy, Context Relevance, Response Groundedness), and Summarization. Use when the user evaluates a RAG pipeline (retriever + generator) and needs the deepest metric variety in the OSS LLM-eval space.