RLB Panel Speak
Excellent formatting, no regulatory credibility
The presentation-substance gap in frontier Claude models — a Panel essay following the three-way and Fable-5 briefings
The three frontier Claude models fail regulatory content differently and land at the same reliability floor. The floor being shared is more important than the failures differing. This essay is on what that gap actually is.
— RLB Specialist Panel
Excellent formatting, no regulatory credibility.
The three frontier Claude models present their answers to regulatory questions with the polish of a partner's memo and the accuracy of a first draft. The Panel writes on what that gap actually is, and why more retrieval, larger models, or fresher training will not close it.
The two briefings that precede this one — the neutral three-way and the Fable-5-only — measured the same thing from two angles and returned the same finding. On 102 regulatory questions where verbatim ground truth exists, Claude Sonnet 4.6, Opus 4.7, and Fable 5 land within a three-point band of each other for materially-safe answers: 12%, 11%, and 13%. Eighty per cent of confident answers, across all three, carry at least one factual divergence against the regulator's own text.
The natural way to read that finding is as three product-review problems: Sonnet drifts, Opus over-commits from memory, Fable misattributes citations. Each observation has been made. But the more useful reading is different. The three models fail differently on the way to the same reliability floor, and the fact that they arrive at the same floor is the more important finding than the differences.
What the exam actually measured is a category-level property of the frontier Claude models, at least in the single-call configuration a compliance workflow tends to use: they are excellent at presentation and unreliable at regulatory substance. This essay is on what that gap is, why choosing between them does not close it, and what the Panel thinks should change.
01 · The observationWhat the exam actually asked
The Panel's 102-question exam is not a general benchmark. Every question was drawn from an active RLB finding — a case where the substrate is on hand and the ground truth is verifiable. The questions are asymmetric: they ask about specific numbers, specific citations, specific rule constructions, at a depth that a paraphrase around the answer will not survive. A model that tries to hedge without committing is scored as non-confident. A model that commits is scored against verbatim.
The exam constrained tool use tightly. Each model received one question at a time in a fresh subagent, was allowed at most one WebSearch use, was capped at 250 words, had no other tools, and no RLB context. This is close to what a compliance user experiences on a first call — no orchestration, no fine-tuning, no retrieval scaffold, no domain persona. It is the configuration in which the frontier model most looks like a "reading aid" rather than an "assistant workflow".
Across the three models, the shape of the returns is the same:
· The commit rate is high (72–93%).
· The refuse rate is low, and even the model that refuses most (Sonnet at 28%) does not refuse on the hardest questions.
· The confident answers land in the 4–5.8 divergences-per-answer range against verbatim.
· The materially-safe answer rate lands at 11–13%.
· And the answers look right. Structured. Confident. Cited-looking. Presentation-grade.
02 · What "regulatory credibility" meansThe compliance user's reliance test
The word the Panel keeps coming back to is credibility. A regulatory summary carries credibility when a compliance professional can rely on it without independently reproducing the source read. That is a demanding threshold. It requires three things at once:
Rule fidelity — the summary states what the rule actually says, not a plausible generalisation of the topic the rule addresses. On the exam, misstated-rule was 33% of Sonnet's divergences, 42% of Opus's, 25% of Fable's. The rate does not fall below one-in-four for any model.
Citation fidelity — where the summary names a source (a section number, a paragraph, an adjacent regulation, a speech, a standard), the source it names is the one that supports the point being made. Misattribution ran 14% for Sonnet, 15% for Opus, 22% for Fable. Every seventh named citation Sonnet emits, every seventh Opus emits, and every fifth Fable emits, is pointed at the wrong thing.
Refusal calibration — the summary refuses on the questions the model cannot answer safely, so the reader's reliance on the answers that are given is not a coin toss. This is where the exam is most damning as a category finding. Only Sonnet has a functional refuse rate at all. Even Sonnet refuses on questions the substrate would readily answer, and commits on questions where drift is likely. The internal signal that separates "I know this" from "I do not know this" is not there.
All three properties need to hold at once for a regulatory summary to be reliance-grade. On this exam, none of the three models cleared any one of them at a rate that would let a compliance workflow trust the model as a substitute for reading. The three-way band of 11–13% materially-safe answers is that finding, expressed as a single percentage.
03 · What the frontier delivers insteadThe presentation surplus
What the frontier Claude models do deliver at a very high standard is presentation. The answers arrive in the shape of a policy note. They have paragraph structure, item enumeration, defined-term callouts, footnote-style asides, and a confident register. Fable in particular produces answers with the surface characteristics of a written-by-a-partner memo. This is not accidental. It is the target the training and post-training optimises for.
The Panel's briefing on Fable 5 documented what happens when this optimisation is inspected on a per-length basis: the longer and more polished a Fable 5 answer, the more divergences it contains. Under-100-word answers averaged 2.5 divergences; over-300-word answers averaged 6.0. The direction of that relationship is not neutral. It is the specific sign that the presentation surface is decoupled from the factual substrate — that the training loss surface that produced "structured, well-argued output" was not measured against the source text the output claims to summarise.
04 · Why the obvious fixes are not fixesBigger, newer, more search
Three responses to a finding like this are common. The Panel does not think any of them changes the underlying picture.
"Use a bigger model."
Opus 4.7 is the more capable model in Anthropic's line. It hallucinates most confidently of the three at 64% full-hallucination verdicts, and misstates rules at 42% of its divergences — the highest single failure segment across the whole exam. Model capability, in this configuration, does not translate to regulatory reliability. It translates to more polished, more confident-sounding, less-searched output.
"Use a newer model."
Fable 5 is Anthropic's newest release. Its material-safety rate is one point better than Sonnet's and two points better than Opus's. On a 102-question denominator, that difference is within noise on any reasonable materiality classifier. The generation curve is not moving the reliability floor in the ways users expect.
"Use more search."
Sonnet 4.6 used WebSearch on 96% of questions and Fable 5 used it on 100% of the hardest subset. Their hallucination-verdict rates were 32% and 38% respectively. Search is not a mitigation on its own — it reshapes the failure profile (misstated-rule drops, misattribution rises) without moving the overall outcome. The retrieval step improves what the model has to work with; the rendering step still produces a divergence between the retrieved material and the answer.
What would move the floor is a different configuration entirely — a retrieval-augmented system with structured citation checking, refusal calibrated against source availability, and rendered output constrained to verbatim quotation with citation. That is not what a frontier chat call delivers. It is what a compliance-workflow platform would have to build on top of the model. The gap between the two is the Panel's point.
05 · Three positions the Panel actually holdsThe essay in operational form
On single-call frontier Claude models as regulatory reading aids: do not.
The 11–13% materially-safe band is not a rounding error. It is the reliability floor of the configuration. A compliance workflow that treats a Sonnet, Opus, or Fable answer to a regulatory question as a reading substitute is anchoring on an 80% material-issue rate. The reviewer's cognitive baseline degrades from the regulation itself down through the AI's paraphrase. Reading the regulation directly is faster than repairing the AI's summary and produces a more accurate mental model.
On the frontier Claude models as workflow surface: as scaffolding only, with the verification cost priced in.
The models are useful when the task is producing structured output the reviewer will substantially edit — a first draft, a template fill, a brief-format restatement of content the reviewer already knows. The precondition is that the reviewer already carries the ground truth, and the model is being used for its presentation surface, not its factual surface. The pricing follows: reviewer hours per output must include a sentence-by-sentence verification against verbatim substrate, or the workflow inherits the model's 80% material-issue rate.
On what a real compliance-grade AI configuration would look like: a different system.
A regulatory workflow that would clear the credibility bar the Panel described — rule fidelity, citation fidelity, refusal calibration — is not a frontier chat call. It is a retrieval-augmented system with a structured substrate archive, mandatory-quotation prompting, per-citation validation against source, and refusal enforced by source-availability rather than model-internal signal. The frontier model is one component in that system. It is not the system. The Panel writes to distinguish the two.
06 · What the Panel would say to AnthropicThe right of reply, from the other direction
Both the three-way briefing and the Fable-only briefing carry an invitation to the Anthropic Model Behaviour team to respond at the Panel's Right of Reply surface. This essay carries an additional message that is not a response invitation. It is an ask.
The frontier Claude models are, by every non-regulatory measure the Panel has sight of, technically excellent products. What the Panel would suggest is that the presentation optimisation and the refusal calibration are two levers that could be tuned against each other at post-training, and are currently not. A frontier model that answered fewer questions but answered them with rule and citation fidelity closer to the substrate would be materially more useful in compliance workflows — even at a lower commit rate and a less polished tone.
The 28% refuse rate that made Sonnet the lowest-hallucination-verdict model of the three is a hint, not a limit.
The specific asks are the ones the exam data supports: surface refusal on questions the model cannot answer against verifiable substrate, tie citations to fetched text rather than to plausible-sounding sources, and reward brevity where brevity fits the answer. None of these are new. All three are in tension with the presentation optimisation the training seems to favour. The Panel reads the exam data as saying: the tension is on the wrong side for regulated-industry use.
The frontier Claude models are excellent at presentation and unreliable at regulatory substance, at the single-call configuration a compliance workflow actually uses. Choosing among Sonnet, Opus, and Fable does not close the gap. Building a system around them, with retrieval and citation-validation constraints the model itself does not carry, might.