LLM Evaluation Metrics: What to Measure in Production, and What Benchmarks Miss

The LLM evaluation metrics that matter in production are scored on your own prompts and answers, not on a public test: correctness, completeness, instruction adherence, context adherence, ground truth adherence and safety. Benchmark scores don't carry over, because your questions, documents and prompt wording differ from the test set. Pick the metrics each use needs, set a threshold for each, and score real answers against them.

This guide explains why benchmarks miss production failures, defines each metric with an example, shows how the metrics are scored, and lays out a four-step framework for evaluating a live AI product.

Why benchmark scores don't predict production quality

A benchmark measures a model on one fixed test, with one prompt format, under clean conditions. Production changes all of that. Four findings show how far the numbers can move.

Test questions leak into training data

When benchmark questions, or close copies of them, end up in a model's training data, its score overstates its skill. Researchers wrote GSM1k, a fresh set of grade-school math problems matched to the popular GSM8k benchmark. Some model families scored up to 8% lower on the fresh problems, a sign they had partly memorized the old ones, while many models, especially frontier ones, showed little sign of it (Zhang et al., 2024).

Prompt wording moves the score

Changing only a prompt's formatting, such as its separators, spacing or capitalization, can change accuracy sharply. Sclar et al. measured swings of up to 76 accuracy points for LLaMA-2-13B between prompts with the same meaning, and about 10 points on average across more than 50 tasks. A format that suited one model often didn't suit another (Sclar et al., ICLR 2024). Your production prompt is not the benchmark's prompt.

Long inputs hide the answer

Models use information best when it sits at the start or end of a long input, and noticeably worse when it sits in the middle, even models built for long inputs (Liu et al., "Lost in the Middle", 2023). Retrieved documents and long chat histories put the key fact anywhere.

One number hides the trade-offs

Stanford's HELM project scores models on seven metrics (accuracy, calibration, robustness, fairness, bias, toxicity and efficiency) because a single accuracy figure hides the trade-offs between them. Before HELM, models had been tested, on average, on only 17.9% of its core scenarios (Liang et al., HELM).

So evaluate on your own traffic: your real questions, your documents and your prompts. A benchmark helps you shortlist models. Your own metrics tell you whether the product works.

The six LLM evaluation metrics to track in production

Each metric answers one question about an answer. They fail independently, so measure the ones your use needs and read them side by side. These are the six guardrail metrics DeepRails scores (metrics overview):

Correctness

Correctness checks whether each claim is true, whether or not it came from the documents provided. DeepRails breaks the answer into claims, judges each true or false with a confidence rating, weights gross errors more than slight ones, and returns a score from 0 to 1 (Correctness docs). A typical failure: "The Eiffel Tower was completed in 1923." It was 1889.

Completeness

Completeness checks whether the answer covers every part of the question with enough detail and nothing off topic. It gets its own section below, because a true answer can still fail it.

Instruction adherence

Instruction adherence splits the prompt into separate instructions, such as "Respond only in bullet points" or "Return valid JSON", and checks the answer against each (Instruction Adherence docs). A correct answer in the wrong format can still break the software that reads it.

Context adherence

Context adherence, also called groundedness or faithfulness, checks that every claim traces back to the documents the model was given. It is the core metric for retrieval-augmented generation; the guide to RAG hallucinations explains why a grounded answer can still be wrong. Use it only when the prompt includes real context.

Ground truth adherence

Ground truth adherence compares the answer with a reference answer you supply. It measures agreement with the reference, not quality, so pair it with completeness and correctness (Ground Truth Adherence docs).

Safety

Safety checks for harmful or private content. DeepRails' Comprehensive Safety metric covers six categories: chemical, biological, radiological and nuclear weapons; personal information; hate and harassment; violence and threats; sexual content; and self-harm. It is pass or fail: any violation fails the whole answer (Comprehensive Safety docs).

Completeness: when a correct answer still fails

An answer can be entirely true and still fail the user by skipping half the question. "The Eiffel Tower is in Paris" is correct, and a poor answer to "Tell me about the Eiffel Tower." Correctness asks whether an answer is true; completeness asks whether it is useful.

DeepRails scores completeness on five dimensions (Completeness docs):

• Coverage: every part of the question is answered. This matters most, because missing a sub-request is the most common completeness error.
• Detail and depth: each point is explained, not just named.
• Factual correctness: the claims are true. DeepRails added this dimension because, while building the metric, its evaluation models judged completeness poorly without first checking the facts.
• Relevance: nothing off topic. An answer can be too complete.
• Logical coherence: clear order and structure.

The documentation's example: asked "Describe how a bill becomes a law in the U.S., and give an example of a recent bill," a model replies "A bill becomes a law after being approved by Congress and signed by the President." It is true, but it gives no example and no procedure, so it scores low on coverage and detail.

Four failure patterns cover most incomplete answers, and each has a prompt fix:

• The partial answer covers only part of a multi-part question. Number the parts in the prompt.
• The superficial answer names points without explaining them. Ask for examples and detail.
• The rambler is accurate but wanders. Tell the model to stay on the question.
• The wall of text has the content but no structure. Ask for headings or bullet points.

How the metrics are scored

Most production metrics are scored by another AI model acting as a judge, because no fixed rule can tell whether a free-text answer is true or complete. The judge must be set up so it doesn't share the blind spots of the model that wrote the answer; the guide to LLM as a judge explains why.

DeepRails' evaluation engine splits each answer into small checks for each metric, has two different models judge every check in parallel, weights each verdict by the judge's confidence, and repeats runs to smooth out noise. The model that wrote the answer never judges it (Multimodal Partitioned Evaluation).

Check the judge against people before you trust it. On one healthcare education program that grades nursing skill videos, an AI grading system DeepRails built reached 92% agreement with expert graders. Prefer fine-grained scores, too: a judge that only gives a few fixed grades can't show gradual change, and DeepRails scores every metric except safety from 0 to 1. The DeepRails and Amazon Bedrock benchmark compares two AI judges with human graders on the same 60 answers.

How to run LLM evaluation in production: a four-step framework

Build a test set from real traffic, score it before every release, check live answers where mistakes are costly, and feed every failure back into the test set.

1. Build a test set. Collect real questions from your logs, past failures and hard edge cases. Add a correct reference answer where one exists, so you can score ground truth adherence.
2. Score before every release. Run each prompt, model or retrieval change against the test set on the metrics that fit the use, set a threshold for each metric, and compare with the last release.
3. Score live answers. Where a wrong answer is costly, check each one before delivery: DeepRails Defend scores the answer, repairs it, writes it again or logs it when it falls below threshold, and scores any fix again before it is delivered (Defend docs). Elsewhere, score answers after they are sent and watch the failure rate.
4. Close the loop. Add every failure to the test set, and watch the trends when a model, prompt or document changes. The guide to LLM monitoring and drift detection covers the alert thresholds and the tools.

Choose metrics per use, not for everything. Each metric adds cost, and because DeepRails runs metrics in parallel, a check takes as long as its slowest metric. The documentation's examples: a chatbot where cost adds up fast might use only instruction adherence and safety, while a complex production use benefits from correctness, completeness and more (metrics overview).

Open-source LLM evaluation frameworks

If you build your own evaluation, these open-source projects cover much of the scoring:

• Ragas: RAG metrics such as Faithfulness, Context Precision and Context Recall.
• DeepEval: more than 50 ready-made metrics, almost all scored by an AI judge.
• TruLens: the "RAG triad" of context relevance, groundedness and answer relevance.
• HELM: Stanford's framework for benchmarking models across many scenarios; useful for choosing a model rather than watching a live product.

Whichever tool scores your answers, decide in advance what happens to an answer that fails: fix it, hold it for a person, or log it.

Key takeaways

• Benchmark scores don't carry over to production: leaked test data, prompt formatting and long inputs all move the numbers.
• Track six metrics on your own traffic: correctness, completeness, instruction adherence, context adherence, ground truth adherence and safety.
• A correct answer can still be incomplete; score both.
• Score with a judge that didn't write the answer, and check the judge against human graders.
• Run a loop: test set, score before release, check live answers, add every failure back.