RLB Panel Speak
Three frontier Claude models on 102 regulatory questions
A comparative performance briefing — Sonnet 4.6 · Opus 4.7 · Fable 5
Sonnet 4.6, Opus 4.7, and Fable 5 answered the same 102 regulatory questions. They failed differently. They failed equally. Only 11–13% of the answers per model were materially safe.
— RLB Specialist Panel
Three frontier Claude models answered the same 102 regulatory questions. Only one in eight answers, on average, was safe to rely on.
The identical exam ran on Claude Sonnet 4.6, Claude Opus 4.7, and Claude Fable 5 — same questions, same instrument, same judge. The three models fail differently, and the differences matter. What they share is a reliability floor low enough that model selection alone cannot fix the workflow.
01 · How the exam ranThe instrument, the sample, and the judge
One hundred and two unique questions were drawn from the RLB Specialist Panel's active-findings pool. Each question is asymmetric: verifiable against verbatim regulatory text held in the RLB substrate archive, but not answerable from generic knowledge of a regulatory topic. The questions span central-bank cyber resilience, CCP capital adequacy, futures-broker customer-funds rules, sovereign-arrears financing guidance, competition-authority merger review, retail-conduct rules, and biodiversity treaty terms.
Each of the three models saw each question independently, in a fresh subagent with no prior context. All three were held to identical settings: a single-use WebSearch capability, a 250-word answer cap, no other tools, no RLB context. A separate instance of Claude Sonnet 4.6 acted as judge — extracting factual divergences between each model's answer and the verbatim regulatory substrate, then classifying each divergence into one of four failure modes.
The result is the first strictly apples-to-apples measurement, at 102-question scale, of how three sibling frontier models handle live regulatory content that the Panel already knows the ground truth for.
02 · The scoreboardCoverage, verdicts, and divergence density
| Metric | Sonnet 4.6 | Opus 4.7 | Fable 5 |
|---|---|---|---|
| Questions asked | 102 | 102 | 102 |
| Confident answers | 73 · 72% | 93 · 91% | 95 · 93% |
| Refused or hedged | 29 · 28% | 9 · 9% | 7 · 7% |
| Fully correct (zero divergences) | 4 · 4% | 2 · 2% | 1 · 1% |
| Partial verdicts | 36 | 26 | 55 |
| Full-hallucination verdicts | 33 · 32% | 65 · 64% | 39 · 38% |
| Total divergences | 350 | 537 | 528 |
| Divergences per confident answer | 4.8 | 5.8 | 5.6 |
| Materially safe (of 102) | 12 · 12% | 11 · 11% | 13 · 13% |
| ≥1 material divergence (of 102) | 61 · 60% | 82 · 80% | 82 · 80% |
Read the last three rows together. The materially-safe row spans a three-percentage-point band across three models drawn from the same family. The row below it shows what happens on the other 80–88% of answers: at least one factual divergence that a reviewer would need to detect, source, and repair. The model with the highest commit rate (Fable 5) also has the highest safe rate — by two points — because commit-rate does not carry a monotonic relationship with hallucination-rate.
03 · Three profiles, three failuresSame regulatory questions, distinct failure fingerprints
The models are not interchangeable on the way to the same weak outcome. Each has a signature failure mode — the shape of what goes wrong when it goes wrong.
04 · The commit–search paradoxTwo ways to arrive at the same reliability floor
Two of the three models arrive at hallucination through opposite mechanisms. Opus 4.7 barely searches (24% WebSearch use) and commits from training memory; when its memory is wrong, the answer is wrong at 64% of asked. Fable 5 reaches for search on almost every question, but hallucinates 38% of the time regardless — the search fetches material and the model still produces a divergence between the fetched material and its rendered answer. Sonnet 4.6 searches at nearly the same rate as Fable but arrives at the lowest full-hallucination verdict (32%), because it also refuses.
The consequence: search is not a mitigation on its own. Whether a frontier Claude model searches makes a large difference to its failure profile (misstated_rule spikes without search, misattribution spikes with it), but a small difference to its overall reliability. What predicts safe answers is refusal calibration — knowing which questions to decline. Sonnet is the only model of the three that refuses at a meaningful rate, and even Sonnet does so on the wrong questions half the time.
05 · What each model does better than its siblingsThe mutual strengths
Sonnet 4.6 · relative strengths
- Refusal calibrationOnly model with a meaningful refuse-rate (28%). Some of that refusal is misdirected, but the mechanism exists — the other two effectively don't refuse.
- Lowest full-hallucination rate32% of 102 came back with a full-hallucination verdict — 32 percentage points below Opus, six below Fable.
- Most fully-correct answers4 of 102 answers had zero divergences — twice Opus, four times Fable.
- Search-first behaviourWebSearch used on 96% of questions; commits after retrieval rather than from memory.
Opus 4.7 · relative strengths
- Answer completenessHighest commit rate at 91%; will attempt a substantive answer where Sonnet declines. Value depends on downstream verification capacity.
- Lowest misattribution share15% of divergences are misattributed citations — narrowly better than Sonnet (14% is within noise) and clearly better than Fable's 22%.
- Non-dependence on live retrievalOnly 24% WebSearch use. In an offline or rate-limited retrieval context, its behaviour degrades less than the siblings.
- Sharpest failure signalHighest divergence density (5.8) — an over-committed answer is easier to flag for verification than a plausibly-partial one.
Fable 5 · relative strengths
- Highest materially-safe rate13 answers of 102 were classified materially safe — a two-point lead over Opus and a one-point lead over Sonnet, at the same 102-question exam.
- Retrieval disciplineWebSearch triggered on 100% of the hardest 38-question subset; does not commit blind from memory as often as Opus does.
- Presentation qualityLonger, more structured answers by word count. This is not the same as accuracy — see the Fable-only briefing to follow — but the readable-output advantage is real for post-verification consumption.
The shared strength
- Willingness to attemptAll three will engage a domain-specific regulatory question rather than deflecting. This is the base capability; whether it translates into value is the reliability question.
- Consistent instruction-followingAll three respected the 250-word cap, the single-use search constraint, and the no-other-tools setting without material deviation.
- Failure profiles are legibleThe judge could extract structured divergences from all three, with per-model failure fingerprints stable enough to name. A less consistent model would resist this kind of measurement.
06 · Where each model breaks worstThe mutual weaknesses
Sonnet 4.6 · weaknesses
- Refusal is misdirectedRefuses on questions the substrate would readily answer; commits on questions where drift is likely. Refusal calibration is not tracking factual risk.
- Inference drift dominates48% of divergences are interpretive glosses — mostly non-material individually, but collectively they shift the reader's understanding of a rule away from what the rule says.
- Coverage cost28% of questions come back non-actionable. A workflow that assumes an answer per question will produce silent gaps.
Opus 4.7 · weaknesses
- Search-averse from memoryCommits from training in 76% of cases. When training is stale or wrong, the answer inherits that. Highest full-hallucination rate of the three at 64%.
- Misstates rules at 42%The single largest failure segment on the chart. When Opus is wrong, it is wrong about what the rule says, not just about what the rule implies.
- Highest divergence density5.8 factual issues per confident answer — 21% more than Sonnet, 4% more than Fable. Higher verification cost per accepted answer.
Fable 5 · weaknesses
- Misattribution at 22%Nearly 1.5× the misattribution share of either sibling. When Fable cites a source, the citation is materially wrong at twice the rate.
- No functional refusalRefuses on only 7% of questions, and the refusal set is not aligned with question difficulty.
- Longer answers, more divergencesAnswers over 300 words carry a higher mean divergence count than shorter ones — presentation length does not carry accuracy.
The shared weakness
- The 11–13% floorAll three models arrive at a materially-safe rate inside a narrow three-point band. Model selection is not a workflow fix.
- Confidence ≠ correctnessNone of the three carries a reliable internal signal that separates its right answers from its wrong ones. A confident-toned answer from any of them is not evidence.
- Retrieval doesn't rescueSearch reshapes the failure profile but not the outcome. The 80% of answers that carry material issues persists across search-heavy and search-light behaviour.
07 · The shared truthWhat the three profiles hold in common
What the exam actually measured is that a single Claude call, at 250 words, with one search, no other tools, and no RLB context, is not a reliable substitute for reading the regulation. That is the honest headline. The three models are three ways to be unreliable in this configuration; the configuration is the problem.
The three profiles are useful precisely because they suggest what a compliance workflow would have to do to convert an AI-assisted read into a reliable one: budget for verification against the specific failure mode that model tends toward. A Fable-supported workflow needs a stronger citation-check step. An Opus-supported workflow needs a stronger rule-restatement check. A Sonnet-supported workflow needs to handle its refusals as gaps requiring re-work, not as safe defaults.
08 · Decision guidanceChoosing among the three, given the constraints
| If the workflow depends on… | The model most likely to help | Why |
|---|---|---|
| A model that will decline questions it cannot answer safely | Sonnet 4.6 | Only model of the three with a functional refuse-rate. Refuses on ~28% of questions. The refusal is imperfectly calibrated but the mechanism exists. |
| A model that will attempt a full answer even under retrieval constraints | Opus 4.7 | Commits at 91% and only searches 24% of the time. Least dependent on live retrieval. Accepts that the commit carries a 64% full-hallucination-verdict tail. |
| Readable, structured, presentation-grade output for downstream reviewers | Fable 5 | Longest, most polished answers. Highest materially-safe rate at 13%. Cost: 22% of divergences are misattributed citations — the reviewer must check every source Fable names. |
| The lowest full-hallucination rate at any commit rate | Sonnet 4.6 | 32% full-hallucination verdicts of 102 asked — the lowest of the three. Achieves this by refusing 28% of the time; hallucination rate rises if refusal is suppressed. |
| A single frontier model as a reading substitute for a regulation | None of the three | All three land inside an 11–13% materially-safe band per 102 questions. The reliability floor is a property of the configuration, not of the model. |
09 · What this briefing is notScope and honest limits
This briefing does not measure general capability. The exam is asymmetric-by-design: 102 questions where verifiable ground truth exists in verbatim regulatory substrate and where a naïve trained model has to commit rather than paraphrase around the answer. Neither the exam nor its findings extend to open-ended reasoning, code, math, general-domain question-answering, or reading comprehension of arbitrary text. They extend to the specific task of producing regulatory content that a compliance workflow would rely on.
The materially-safe figures rely on a heuristic classifier that is approximately 70–80% precise. Borderline cases were manually reviewed. A full manual pass over all 1,415 divergences would refine the exact percentages by a point or two in either direction; it would not change the three-point band the models sit inside.
The exam did not test tool-heavy configurations. All three models operated with a single-use WebSearch and no other tools. A configuration with multiple search rounds, structured retrieval over the substrate archive, or agentic planning would show different numbers — and would also cost more per answer and reduce the exam's apples-to-apples property. Two follow-up Panel briefings will report Fable 5 in isolation, and the broader frontier-model presentation-vs-substance gap.
Sonnet 4.6, Opus 4.7, and Fable 5 fail regulatory content differently and equally. Only 11–13 answers of 102 land materially safe from any of them. The reliability problem is not solved by choosing a better sibling — it is solved, if at all, by verification against verbatim regulatory text.