DeepRails vs Amazon Bedrock: Hallucination Scoring Benchmark and Bedrock Guardrails' Limits

We scored 60 AI answers with DeepRails and with Amazon Bedrock's built-in AI judge, then compared both with scores from human graders. On answers people graded as partly right (50 or 75 out of 100), DeepRails was off by 7 to 8 points on average; Bedrock was off by 17 to 22. On answers that were clearly right or clearly wrong, Bedrock did as well or better.

This article gives the method, every result and the raw data. It also covers what Amazon Bedrock Guardrails, AWS's runtime filter, can and can't check, what each costs, and how teams move from blocking bad answers to correcting them.

1. Why scoring accuracy matters in production

Every AI guardrail system makes two types of errors: false positives (flagging good outputs as hallucinations) and false negatives (letting bad outputs through). Both are expensive in production.

False positives create unnecessary remediation cycles, increase latency, and waste compute. False negatives are worse: they deliver hallucinated content to users, erode trust, and in regulated industries, create legal and compliance exposure.

The hard cases are in the middle. A clearly wrong answer and a clearly right one are easy for any judge. Many production failures look like an answer that is mostly right with one wrong claim, or complete except for the one step the user asked about. Most DeepRails customers set their pass/fail thresholds between 40% and 80% (docs), so a judge that is accurate there decides whether the threshold works.

2. How we tested

Every input, output, human rationale and individual score is in the raw data (Google Sheet, view only). The study is DeepRails' own, not an independent audit.

3. Benchmark results

Average distance from the human score, in points out of 100, by how the human graded the answer. Lower is better; for each metric, the better of the two is green.

4. What this means in practice

A threshold is only as good as the score behind it. If you block or repair every answer that scores under 70%, a judge that can only answer 0, 50 or 100 puts a 75 answer on the wrong side of the line every time it says 50.

5. What Amazon Bedrock Guardrails can and can't check

Bedrock Guardrails is AWS's runtime filter for AI inputs and answers. It has six safeguards, all generally available (AWS documentation, read October 23, 2026):

• Content filters for text and images: hate, insults, sexual content, violence, misconduct, and prompt attacks such as jailbreaks and prompt injection.
• Denied topics you define, and word filters.
• Sensitive information filters for personal data and your own patterns, which can mask what they find.
• Contextual grounding checks, which give each answer a grounding score and a relevance score from 0 to 1 against a source and query you supply; you set the pass mark between 0 and 0.99 (AWS).
• Automated Reasoning checks, which test an answer against logic rules you write and return findings such as valid or invalid.

Guardrails works with models outside Bedrock too, including OpenAI and Google Gemini models, through its ApplyGuardrail API (AWS). What it doesn't do: none of the six safeguards scores whether an answer is complete, and only contextual grounding (against your source) and Automated Reasoning (against your rules) check whether it is true. Bedrock's correctness and completeness scoring lives in its separate model-evaluation tool, the judge tested above.

How DeepRails scores: Multimodal Partitioned Evaluation

DeepRails uses Multimodal Partitioned Evaluation (MPE), the engine behind every DeepRails metric (docs).

• Partitioned reasoning: Each output is decomposed into verifiable units. Claims are extracted and evaluated individually rather than assessing the entire output as a monolithic block.
• Dual-model consensus: Two different LLMs (often from different providers) evaluate each unit in parallel. The generating model never judges itself.
• Confidence calibration: Each judge self-reports confidence for every sub-check. Final scores aggregate using confidence-aware weighting, so low-confidence evaluations carry less weight.
• Multiple evaluation runs: Evaluations run several times per model and are averaged to reduce noise.
• Continuous scoring: Results are returned as scores from 0 to 1 per metric, so teams can set thresholds in small steps. Comprehensive Safety is the exception: any violation fails it.

6. Beyond detection: the correction gap

Accuracy is one dimension of the comparison. The other is what happens after a hallucination is detected.

FixIt: iterative output improvement

FixIt takes the original output and the failure rationale (which specific claims failed, which metrics were below threshold, and why) and iteratively improves the response. It preserves what was correct and rewrites what failed, so it suits answers that are mostly right and outputs whose format must stay the same.

ReGen: complete regeneration with controlled variance

ReGen generates a completely new response from the original prompt when the initial output is too flawed for iterative repair. The new output is evaluated against the same guardrails before delivery. Best suited for cases where the original response has fundamental structural or factual problems.

7. Cost and latency comparison

There are real trade-offs in accuracy, cost, and speed. Prices below are the providers' list prices on October 23, 2026.

Bedrock Guardrails costs far less per check. For high-volume, low-stakes screening where cost matters more than nuance, that is a legitimate advantage.

DeepRails costs more per call because two judges score every claim. The trade-off: finer scores, a rationale for every metric, and automatic correction. For outputs in healthcare, legal, finance, or customer-facing products where a wrong answer costs more than the check, that trade usually pays. Teams can also screen with Super Fast and reserve Precision Max for critical paths.

8. When to use each

This is not a one-size-fits-all decision. Both systems have valid use cases, and many teams run both.

9. Practical migration guide

For teams using Bedrock Guardrails and adding DeepRails, the integration patterns are similar: both are API checks that sit between your application and the end user. You can keep Bedrock's filters and add DeepRails for scoring and correction.

# Python
pip install deeprails

from deeprails import DeepRails
client = DeepRails(api_key="your-api-key")

# Create a Defend workflow
workflow = client.defend.create_workflow(
    name="production-guardrails",
    threshold_type="automatic",
    improvement_action="fixit",
    automatic_hallucination_tolerance_levels={
        "correctness": "high",
        "completeness": "medium",
        "comprehensive_safety": "high"
    }
)

# Submit an event for evaluation + correction
event = client.defend.submit_event(
    workflow_id=workflow.workflow_id,
    model_input={"user_prompt": "Explain HIPAA requirements..."},
    model_output="HIPAA requires...",
    model_used="gpt-6.1-sol",
    run_mode="precision"
)

10. Getting started

If your production AI needs accurate scores on partly right answers, automatic correction, and fine control over thresholds, DeepRails is built for that problem. Test it on your own data first.

See DeepRails in production