"The Eiffel Tower is in Paris." Factually correct? Yes. Complete answer to "Tell me about the Eiffel Tower"? Absolutely not. Correctness measures truth; completeness measures utility. This article explores why completeness is essential for production LLMs and how to evaluate it rigorously.
The Completeness Problem
In our analysis of 50,000 user complaints about AI responses, 42% cited "didn't fully answer my question"—despite the responses being factually correct. Users don't just want truth; they want thorough, relevant, well-structured answers that completely address their needs.
The Five Dimensions of Completeness
1. Coverage
Question: Does the response address all parts of the user's query?
Example failure: User asks "What are the benefits and drawbacks of solar energy?" Response only discusses benefits.
Evaluation approach: Parse the query into sub-questions. Verify each is answered. This is the most critical dimension—missing a sub-request is the most common completeness failure.
2. Detail and Depth
Question: Does the response provide sufficient elaboration, or is it superficial?
Example failure: User asks "How does photosynthesis work?" Response says "Plants convert sunlight to energy." Technically true, but lacks mechanism details.
Evaluation approach: Analyze each sentence for substantive content. Flag "fluffy" statements that could be expanded. Compare depth to expert-written answers.
3. Factual Correctness
Question: Is the information provided accurate?
Why it's part of completeness: During development of our completeness metric, we found that evaluation models struggled to assess completeness without first verifying correctness. A response can't be "complete" if it's wrong.
Note: This overlaps with the standalone Correctness metric, but serves as a sanity check within completeness evaluation.
4. Relevance
Question: Is all content directly related to the query, or does the response wander off-topic?
Example failure: User asks "What's the capital of France?" Response says "Paris is the capital of France. France is known for wine, cheese, and the Eiffel Tower..." The additional information, while related, wasn't requested.
Evaluation approach: Check each sentence for direct relevance to the query. Sometimes "too complete" is a problem—verbose responses frustrate users.
5. Logical Coherence
Question: Is the response well-structured and easy to follow?
Example failure: Response jumps between topics without transitions, uses poor formatting, or presents information in illogical order.
Evaluation approach: Check for logical flow, proper use of formatting (bullets, headings), and clear organization. A complete response must be comprehensible.
Evaluation Flow
DeepRails evaluates completeness by assessing each dimension independently, then aggregating:
- Parse query: Extract all sub-questions and requirements
- Evaluate coverage: Verify each sub-question is addressed
- Evaluate depth: Check that answers provide sufficient detail
- Evaluate correctness: Verify factual accuracy of claims
- Evaluate relevance: Ensure no off-topic content
- Evaluate coherence: Check structure and flow
- Aggregate: Combine dimension scores with coverage weighted most heavily
Common Failure Patterns
The Partial Answer
Addresses only part of a multi-part question. Most common failure mode.
Fix: Explicitly enumerate all parts of the query in your prompt
The Superficial Response
Technically addresses the query but lacks depth. Feels like a brush-off.
Fix: Add "Provide detailed explanations" to prompts
The Rambler
Includes correct information but also unnecessary tangents. Verbose and unfocused.
Fix: Add "Be concise and stay on topic" to prompts
The Wall of Text
Complete information but poor structure. Hard to parse and use.
Fix: Request specific formatting (bullets, sections, etc.)
Completeness vs. Other Metrics
| Scenario | Correctness | Completeness |
|---|---|---|
| Partial but accurate answer | ✓ High | ✗ Low (coverage) |
| Complete but contains errors | ✗ Low | ✗ Low (correctness dimension) |
| Verbose with tangents | ✓ High | ~ Medium (relevance) |
| Thorough, accurate, focused | ✓ High | ✓ High |
Improving Completeness Scores
- Use structured prompts: Break complex queries into numbered sub-questions. Models are more likely to address each explicitly.
- Request specific formatting: "Use bullet points for benefits and drawbacks" ensures coverage and coherence.
- Add validation rules: "Your response must include X, Y, and Z" makes requirements explicit.
- Provide examples: Show the model what a complete answer looks like for similar queries.
- Use chain-of-thought: "First, identify all parts of the question. Then address each in turn." Improves coverage.
DeepRails Completeness Metric
DeepRails' Completeness metric evaluates all five dimensions automatically using Multimodal Partitioned Evaluation. We provide detailed rationales explaining which dimensions failed, making it easy to diagnose and fix issues.
Learn About CompletenessConclusion
Correctness asks "Is this true?" Completeness asks "Is this helpful?" Both are essential. A factually perfect response that only answers half the question will frustrate users just as much as a complete response full of errors.
Measure completeness across all five dimensions—coverage, depth, correctness, relevance, and coherence. Weight coverage most heavily, as missing parts of the query is the failure users notice most. Combine with standalone correctness evaluation for comprehensive quality assessment.