Most RAG evaluation systems make a fatal error: they conflate grounding (does the response use the provided context?) with accuracy (is the response factually correct?). This conflation causes two catastrophic failure modes—blocking correct outputs that add helpful context, and passing incorrect outputs that are technically "grounded" in flawed source documents. This article establishes the critical distinction and provides a framework for measuring each independently.
The Core Problem
A response can be grounded but wrong (source document contains errors), or correct but ungrounded (model adds accurate information from pre-training). Treating these as the same metric means you can't diagnose which failure mode is occurring—and you'll apply the wrong fix.
Defining the Distinction
Context Adherence (Grounding)
Question: Can every claim in the response be traced back to specific passages in the provided context?
Passing example: Context says "Q3 revenue was $10M." Response says "The company earned $10M in Q3." ✓ Grounded
Failing example: Context says "Q3 revenue was $10M." Response says "The company earned $10M in Q3, continuing its strong growth trajectory." ✗ "Strong growth trajectory" not in context
Correctness (Factual Accuracy)
Question: Are the claims in the response verifiably true according to authoritative sources or world knowledge?
Passing example: Response says "The Eiffel Tower is in Paris." ✓ Factually correct (regardless of whether context mentioned it)
Failing example: Response says "The Eiffel Tower was completed in 1923." ✗ Factually wrong (actual: 1889)
The Four Quadrants
Responses fall into four categories based on these independent dimensions:
✓ Grounded + ✓ Correct
Ideal state. Response uses only provided context, and that context is accurate.
Example: Context says "Paris is the capital of France." Response says "Paris is the capital of France."
✗ Grounded + ✓ Correct
Helpful extrapolation. Response adds accurate information beyond context. May or may not be desirable depending on use case.
Example: Context says "Paris is the capital." Response says "Paris is the capital of France, known for the Eiffel Tower."
✓ Grounded + ✗ Correct
Source document error. Response faithfully reproduces incorrect information from context. Grounding is working; source quality is the problem.
Example: Context says "The Eiffel Tower was completed in 1923." Response repeats this error.
✗ Grounded + ✗ Correct
Hallucination. Response invents information not in context, and that information is wrong. Worst case.
Example: Context says "Paris is a city." Response says "Paris is the capital of Germany."
Why the Distinction Matters: Diagnostic Power
Measuring these dimensions separately tells you what's broken and how to fix it:
| Failure Pattern | Root Cause | Fix |
|---|---|---|
| Low grounding, high correctness | Model ignoring context, using pre-training | Strengthen grounding instructions; reduce temperature |
| High grounding, low correctness | Source documents contain errors | Improve retrieval quality; add fact-checking layer |
| Low grounding, low correctness | Model hallucinating freely | Add strict grounding guardrails; consider model change |
Use Case: When to Prioritize Which Metric
Prioritize Context Adherence When:
- Legal/compliance contexts: Responses must be defensible based on cited documents. Adding external information creates liability.
- Customer support: Agents must follow official policies exactly. Helpful extrapolation can contradict company guidelines.
- Research summarization: Users need to know what the paper actually said, not what the model thinks is true.
Prioritize Correctness When:
- Medical/safety-critical: Wrong information is dangerous regardless of source. Verify against authoritative knowledge bases.
- Educational content: Teaching incorrect information is worse than teaching incomplete information.
- Fact-checking applications: The whole point is verifying truth, not source fidelity.
Require Both When:
- Financial reporting: Must cite specific documents AND be factually accurate.
- Regulated industries: Compliance requires both source traceability and factual correctness.
- High-stakes decision support: Users need confidence that responses are both grounded and true.
Implementation: Measuring Each Dimension
Context Adherence Evaluation
- Extract claims from the response (atomic factual statements)
- Map each claim to specific passages in the context
- Score = (mapped claims) / (total claims)
- Flag unmapped claims for review or correction
Correctness Evaluation
- Extract claims from the response (same as above)
- Verify each claim against authoritative knowledge bases or fact-checking APIs
- Score = (verified claims) / (total claims)
- Flag false claims regardless of whether they appeared in context
DeepRails Measures Both Independently
DeepRails provides separate Context Adherence and Correctness metrics. You can configure workflows to require one, both, or neither depending on your use case. Our evaluation engine automatically handles claim extraction, mapping, and verification.
Common Pitfalls
- Using a single "RAG quality" metric: This hides whether failures are grounding issues or source quality issues.
- Penalizing helpful extrapolation: If your use case allows it, don't punish models for adding accurate context.
- Ignoring source quality: High grounding scores mean nothing if your source documents are wrong.
- Over-constraining grounding: Requiring 100% grounding can make responses robotic and unhelpful. Find the right threshold for your domain.
Conclusion
Context adherence and correctness are orthogonal dimensions of RAG quality. Measuring them separately provides diagnostic power that a single combined metric cannot. You'll know whether to fix your retrieval system, your source documents, or your grounding instructions.
The right balance depends on your use case. Legal applications demand strict grounding; educational applications demand strict correctness; regulated industries demand both. Define your requirements explicitly, measure each dimension independently, and optimize accordingly.