Overview
Langfuse is a production LLM observability platform (per lf-gh (opens in new window)).
Step 1 - Install
Per lf-gh (opens in new window):
pip install langfuseFor TypeScript:
npm install @langfuse/tracingSet up project credentials per Langfuse self-hosted or cloud project (LANGFUSE_PUBLIC_KEY + LANGFUSE_SECRET_KEY + LANGFUSE_HOST).
Step 2 - Instrument with @observe
Per langfuse.com/docs/sdk/python/decorators (opens in new window):
Python:
from langfuse import observe
@observe(name="llm-call", as_type="generation") # auto-captures inputs, outputs, timings, errors
async def my_async_llm_call(prompt_text):
return "LLM response"TypeScript - create an observation, do work inside it, then end it:
import { startActiveObservation } from "@langfuse/tracing";
const { observation, end } = startActiveObservation({
name: "llm-call",
type: "generation",
input: { prompt: promptText },
});
try {
const result = await callMyLLM(promptText);
observation.update({ output: result });
return result;
} finally {
end();
}Validation: After your first instrumented call, open the Langfuse UI → Traces. You should see a new trace with nested observations, inputs, outputs, and timings. If no trace appears within ~30 s:
Step 3 - Update current observation with metadata
Per lf-py-deco (opens in new window):
from langfuse import get_client
langfuse = get_client()
with langfuse.start_as_current_observation(as_type="span", name="data-processing"):
langfuse.update_current_span(metadata={"step1_complete": True})Common metadata fields used in production:
Step 4 - Score traces
Per langfuse.com/docs/scores (opens in new window):
langfuse.score(
trace_id="...",
name="answer_relevance",
value=0.87, # numeric (0-1); also supports categorical (string) and boolean
comment="Judged by GPT-4 rubric"
)Scores can come from:
Validation: After scoring, open the Langfuse UI → Traces → select the trace. The score should appear in the Scores panel. If missing, confirm the trace_id matches an existing trace and that the score call did not raise an exception.
Step 5 - Datasets for offline eval
Datasets are collections of (input, expected_output) items built from production traces, CSV / JSONL imports, or the UI. Run one against your app to diff vs a baseline:
items = langfuse.get_dataset_items(dataset_id="...")
for item in items:
actual = my_llm_app(item.input)
item.run(actual) # links the run back to the dataset for diff vs baselineSee references/datasets.md for building datasets from production traces, the lf-ds (opens in new window) API signature, and the run-validation checkpoint.
Step 6 - Prompt management
Fetch the current production prompt at runtime and pin its version so prompt drift is traceable:
from langfuse import get_client
langfuse = get_client()
# Fetch the production-labelled prompt; falls back to cache if offline
prompt = langfuse.get_prompt("my-prompt-name", label="production")
compiled = prompt.compile(user_input=user_query)
# Pass compiled text to your LLM call
response = my_llm_call(compiled)Iterate prompt text in the Langfuse UI and roll out new versions per environment (production / staging labels) without code deploys. Version history and A/B comparison are available in the UI.
Validation: After an instrumented call using a managed prompt, open the Langfuse UI → Traces → select the trace. The prompt name and version should appear in the trace metadata, confirming prompt version is pinned and attributable.
Step 7 - CI integration
Langfuse is observability-side, not pre-deploy CI-side. Post-deploy CI patterns query the Langfuse API for recent traces and assert on aggregate metrics (a score-query gate, eval-on-trace, cost-regression, score-based alerting). Runnable gate script and the full pattern list: references/ci-integration.md.
Anti-patterns
| Anti-pattern | Why it fails | Fix |
|---|---|---|
| Trace everything in production with no sampling | Cost explodes at scale | Use level=DEBUG + UI-side sampling (Step 3) |
| Score traces only via UI (no automated path) | Can't catch silent regressions | Automated langfuse.score() per trace (Step 4) |
| Pull production trace inputs without privacy review | PII leakage into eval datasets | Cross-ref synthetic-pii-generator for fixture sanitization before promotion |
| Skip prompt versioning | Prompt drift breaks attribution | langfuse.get_prompt() with version pin (Step 6) |
| Conflate Langfuse with pre-deploy eval | Tries to be both; wins neither | Pair Langfuse (post-deploy) with Promptfoo/DeepEval/Ragas (pre-deploy) |
Version notes
Per lf-gh (opens in new window), the SDK was rewritten in v4 and released in March 2026; this skill targets the v4 API. v3 patterns are no longer supported - pin the SDK version in requirements, and for v3 codebases see the upstream migration guide.
Limitations
References
Langfuse is observability-side, not pre-deploy CI-side. Typical post-deploy CI patterns use the Langfuse API to query recent traces and assert on aggregate metrics.
Score-query gate
import httpx, os, sys
from datetime import datetime, timedelta, timezone
LANGFUSE_HOST = os.environ["LANGFUSE_HOST"]
headers = {"Authorization": f"Bearer {os.environ['LANGFUSE_SECRET_KEY']}"}
# Fetch average answer_relevance score over the last hour
one_hour_ago = (datetime.now(timezone.utc) - timedelta(hours=1)).isoformat()
resp = httpx.get(
f"{LANGFUSE_HOST}/api/public/scores",
params={"name": "answer_relevance", "fromTimestamp": one_hour_ago},
headers=headers,
)
scores = resp.json()["data"]
avg = sum(s["value"] for s in scores) / len(scores) if scores else 1.0
if avg < 0.75:
print(f"answer_relevance regression: {avg:.2f} < 0.75 threshold")
sys.exit(1)Validation: After the CI job runs, confirm the script exits 0 and that the queried score window covers the expected deployment window. If scores is empty, verify the fromTimestamp range and that instrumented calls have been scored in that period.
Other CI wiring patterns
Langfuse datasets are collections of (input, expected_output) items. Build them from three sources:
Shipping traces from production into a dataset, then replaying them, is the core offline-regression workflow: item.run() links each replay back to the dataset so the Langfuse UI can diff the run against a baseline. See langfuse.com/docs/datasets (opens in new window) for the current API signature.
Validation
After running a dataset, open the Langfuse UI → Datasets → select your dataset. Each item run should appear under the Runs tab linked to its trace. If runs are missing, confirm dataset_id is correct and that item.run() did not raise an exception.