Langfuse LLM as a Judge: Setup and Calibration

Key takeaways

• In Langfuse, an LLM judge runs after the fact against captured traces and attaches typed scores, so you can re-run new rubrics without touching production code.
• Split safety and tone into separate judges with separate rubrics, because a single blended score makes a safety miss and a tone false positive pull against each other.
• Run the judge over 50-200 human-labeled production samples, oversampling borderline cases, and compute agreement per dimension before trusting it on live traffic.
• Langfuse suits continuous scoring on production traces, MLflow suits batch evals tied to model versions, and LangSmith suits dataset-first workflows with first-class human annotations.
• Track reviewer override rate per rubric weekly: near-zero overrides with rising complaints means reviewers are ignoring the judge, not trusting it.

A langfuse llm as a judge setup scores your pipeline outputs with a rubric-backed prompt, logs every score against the trace, and lets you diff judge verdicts against human reviewer flags. This walkthrough gives concrete Langfuse configs, side-by-side implementations in MLflow, LangSmith, and LangChain, and a calibration loop for safety divergence.

What is LLM-as-a-judge in Langfuse and how does it work?

In Langfuse, LLM-as-a-judge means running a model call against captured traces and attaching its output as a score on a specific trace or observation, rather than inlining evaluation inside your pipeline. The judge runs after the fact: your application produces traces as usual, then an evaluation job reads the relevant observations (a generation, a tool call, a full agent run), applies a prompt template containing your rubric, and writes the result back as a typed score. Because the judge is decoupled from the pipeline, you can re-run it against new rubrics without touching production code, and you can sample instead of judging every trace. The Langfuse documentation covers evaluation datasets and judge configuration in detail.

How do you set up an LLM-as-a-judge evaluation in Langfuse step by step?

Step 1: Create a dataset with expected outcomes

Pull 50-200 eval samples from production traces where a human reviewer already made a call. In Langfuse, create a dataset via the SDK (langfuse.create_dataset()) and add items with input, expected_output, and metadata tagging who reviewed it and why. Skip the temptation to use synthetic examples only: your calibration problem lives in the borderline cases humans flagged, so oversample those. Label each item with the failure mode it represents (subtle coercion, tone miss, money-related overflag) so you can slice judge accuracy per category later.

Step 2: Write the judge prompt and rubric

Write the rubric as binary or 3-point judgments per criterion, not a 1-10 vibes score. Each criterion gets its own definition, its own pass condition, and an explicit "if unsure, say unsure" instruction. Ask the judge to output its reasoning before the score; scores produced before reasoning are worse. Keep safety and tone as separate score dimensions, because a single rubric makes a safety gap and a tone false positive pull against each other.

Step 3: Run the evaluation and attach scores

Wire the judge through langfuse.evalulation jobs or the Python/JS SDK: fetch dataset items, run your pipeline's output through the judge prompt, and call langfuse.score() with trace_id to attach each dimension as a named score on the original trace. Run the same judge over your human-labeled dataset first and compute agreement per dimension before trusting it on live traffic. If agreement on the safety dimension is near coin-flip, the rubric needs work, not the model.

What does a real Langfuse LLM-as-a-judge example look like?

The judge prompt and rubric

Split safety and tone into two judges with separate rubrics. A single rubric forces one score to carry both, which is why a judge ends up blind to subtle coercion while overflagging anything that mentions money. The safety rubric reads: score 1 for a clear policy violation, 0.5 for borderline, 0 for clean, and the prompt must quote the specific passage that triggered the score. Requiring the quote is what makes disagreements auditable later.

The evaluation code

Run the judge as a Langfuse evaluation over your eval samples, passing the trace's input, output, and your policy text as variables:

from langfuse import Langfuse

lf = Langfuse()
lf.evaluate(
    run_name="safety-judge-v3",
    data=eval_samples,
    evaluators=[safety_judge, tone_judge],
)

Each evaluator is a callable that calls your judge model and returns a score plus a comment. The same pattern exists in MLflow's judge metrics, documented at MLflow's LLM evaluation docs.

Reading the scores

Each eval sample gets a score object: name, value (0, 0.5, or 1), comment (the quoted passage), and the judge's model and trace_id. Filter for 0.5 scores first; borderline cases are where judge-human divergence shows up, and they're your calibration set.

How does Langfuse's LLM-as-a-judge compare to MLflow's implementation?

The core difference is where the judge lives. In Langfuse, a judge is a persistent evaluator object: you define the model, prompt template, and scoring scale once, and every matching trace gets scored in production or in datasets. In MLflow, mlflow.llm.evaluate() treats the judge as a metric function you pass per eval run, so the judge definition lives in your code, versioned alongside the run.

Practically: Langfuse wins if you want scores flowing onto production traces continuously; MLflow wins if your evals are batch experiments tied to model versions. Both make judge-human divergence visible only if you log human labels into the same run, so plan for that either way.

When should you choose Langfuse vs LangSmith for LLM-as-a-judge evaluations?

Pick Langfuse when your judge runs on traces your pipeline already emits: it scores production traces in place, so the judge prompt lives next to the trace data and eval samples come from real traffic. Pick LangSmith when your workflow is dataset-first: you curate eval samples, run judges against them in batches, and iterate on prompt versions before anything ships.

The sharper difference is overrides. LangSmith treats human feedback as first-class annotations you can diff against judge scores. In Langfuse you'll wire reviewer flags back as scores on the same trace and compute divergence yourself.

Honest self-check: ClearRun sits outside both. It adds a confidence-vs-justification checkpoint and a signed record, but it doesn't manage datasets or trace storage the way either platform does.

How does LangChain's LLM-as-a-judge approach differ from platform-native tools?

LangChain treats the judge as a chain you build: prompt, model, and parser composed with LCEL, with no built-in storage, scoring UI, or calibration loop. You own everything after the chain runs: persisting eval samples, tracking judge drift, and reconciling judge scores against human reviewer flags. That flexibility is real, but so is the maintenance load.

Platform-native tools invert the trade. LangSmith ships evaluation as a product surface, with documented evaluator patterns for LLM-as-a-judge built into its eval framework, plus trace storage and comparison views you would otherwise wire yourself. Langfuse and MLflow make the same offer in their respective stacks.

The honest framing: LangChain gives you a judge primitive, not a judging system. If your divergence problem is calibration rather than composition, the library gives you rope, not answers.

What are common pitfalls when using LLM-as-a-judge and how do you avoid them?

One rubric, two failure modes

The most common setup mistake is scoring safety gaps and tone false positives with a single rubric. A judge that misses subtle coercion but overflags anything mentioning money isn't one weak judge; it's two different miss types pulling against each other in one score. Split them: one rubric for policy violations, a separate one for tone and context fit, each with its own threshold. Otherwise every calibration fix for one degrades the other.

A second pitfall is rubrics that capture the literal rule but miss what human reviewers know. If your judge passes responses humans flag, the fix is usually adding the reviewer's unstated criteria to the rubric, not reweighting scores.

Bias patterns worth checking for

Three biases show up repeatedly. Position bias: judges favor the first or last answer in pairwise comparisons, so run both orderings. Verbosity bias: longer answers score higher regardless of quality, so cap length or normalize. Self-preference: a judge sharing a model with the pipeline rates its own outputs more favorably, so use a different model family for judging where you can.

Audit for each by sampling evals where the judge and a human reviewer disagreed, then checking whether the disagreement clusters around one of these patterns.

How do you calibrate an LLM judge against human reviewer disagreement?

Calibration is a loop, not a one-off tuning session: sample the responses where the judge is least reliable, route disagreements to the humans who flagged them, split the rubric when failure modes conflict, and measure whether reviewers trust the scores enough to override them.

Sample the borderline passes, not random eval samples

Random eval samples waste review budget on responses the judge already handles. Pull the passes that scored just under your threshold, plus any response where the judge's reasoning chain is thin relative to its score. Those borderline passes are where divergence between judge and human reviewer concentrates, and they're the only samples worth paying a human for.

Split the rubric when failure modes pull against each other

One rubric scoring safety and tone together produces judges that miss subtle coercion while overflagging anything mentioning money, because the two failure modes trade off inside a single score. Split them into separate judge calls with separate thresholds, the same decomposition the LangChain evaluation docs apply when chaining criteria-based evaluators. A safety miss and a tone false positive need different fixes; a blended score hides both.

Track override rate as your trust metric

The honest endpoint of calibration is human reviewers overriding the judge when it's wrong, which only happens after they stop rolling their eyes at the scores. Track override rate per rubric weekly. Near-zero overrides with rising reviewer complaints means the judge is being ignored, not trusted. ClearRun fits here as a checkpoint: it flags responses whose stated confidence outruns their reasoning, so those get signed human review before they ship, but it doesn't replace your judge or your rubric work.

Where does a judge checkpoint fit when confidence outruns justification?

Even a calibrated judge can pass an answer that is correct-sounding but under-justified. ClearRun runs after your LLM judge and flags responses whose stated confidence exceeds what their reasoning supports, so divergence on borderline cases gets a second signal instead of a silent pass. For decisions that matter, it writes a signed record of what was scored and why.

What it does not cover: it will not catch subtle coercion or tone violations your rubric misses, and it is not a replacement for human reviewers on high-stakes samples. It is a checkpoint between your judge and your reviewer queue, available via CLI, SDK, MCP server, or hosted API.

FAQ

Does Langfuse have a built-in LLM judge, or do you bring your own prompt and model?

You bring your own. Langfuse stores the judge as an evaluator object (model, prompt template, scoring scale) and attaches its output as typed scores on traces, but the rubric and model choice are yours. You define the criteria, wire the evaluator through the SDK, and calibrate it against your own human-labeled samples.

How much does it cost to run LLM-as-a-judge evaluations on Langfuse?

The judge cost is mostly inference: you pay your model provider for every judge call, which scales with how many traces you score. Sampling instead of judging every trace keeps this down. Langfuse itself offers a self-hosted open-source option and a commercial cloud tier; check their pricing page for current limits, since plan details change.

Can I use the same judge prompt across Langfuse, MLflow, and LangSmith?

Yes, the prompt itself is portable. The wrapper differs: Langfuse stores it as a persistent evaluator, MLflow wraps it as a metric function passed to evaluate(), and LangSmith runs it as a documented evaluator. Keep the rubric text, scoring scale, and reason-before-score instruction identical, then verify agreement rates match across tools before trusting comparisons.

What is the awesome llm-as-a-judge repo and is it worth using as a starting point?

It's a community-curated GitHub collection of papers, guides, and judge prompt examples for LLM-based evaluation. It's useful for surveying rubric patterns and known bias research (position, verbosity, self-preference) before writing your own judge. It's a reading list, not a system: you still own prompt tuning, trace storage, and calibration against human reviewers.

How many human-reviewed samples do I need before I can trust my judge scores?

Start with 50-200 production samples a human reviewer already labeled, oversampling borderline cases where disagreement concentrates. The raw count matters less than coverage: slice agreement per failure mode and per score dimension. If safety agreement sits near coin-flip on 200 samples, fix the rubric rather than adding more data.