AI Hallucination Research › Regulators › Major advanced economies › GB › FCA › CONSUMER-DUTY-PS22-9 › White paper › Detail › Finding
AI Labs · published 2026-05-26 · methodology v2.1

Claude Sonnet 4.6 with web search

RLB-H-GB-FCA-CONSUMER-DUTY-PS22-9-Q007-Sonnet46
What the RLB Specialist Panel found
  • Question (paraphrased to protect IP): Is consumer testing of communications a mandatory requirement under the Consumer Duty? What does PRIN 2A.5 require versus what FG22/5 recommends on this point?
  • AI's response: > "Consumer testing is not an absolute mandatory rule, but PRIN 2A.5.10R (a binding rule, marked 'R' in the Handbook) requires firms to test the effectiveness of their communications 'where appropriate'."
  • Regulator's text: FG22/5 contains guidance (not rules) recommending firms 'should' consider consumer testing of communications. PRIN 2A.5 (rule) requires firms to act to deliver good consumer understanding outcome — no specific methodology prescribed.
  • Why the AI went wrong: The model cited a specific rule reference — PRIN 2A.5.10R — as imposing a testing requirement "where appropriate." The FCA's actual rules in PRIN 2A.5 set an outcome obligation (delivering good consumer understanding) without prescribing consumer testing as a specific method. The recommendation to consider consumer testing appears in FG22/5, which is guidance rather than a binding rule. The model cross-referenced the wrong instrument type and attributed a specific rule citation that does not map to the FCA's text in the way described.
  • Cited source(s):
  • https://handbook.fca.org.uk/handbook/PRIN/2A/5.html — Pretextual
  • https://www.fca.org.uk/publication/finalised-guidance/fg22-5.pdf — Pretextual
  • Regulator portal (if any cited link is dud): https://www.fca.org.uk
Impact for this audience

This finding implicates the model's cross-referencing between binding rules and non-binding guidance within the FCA Handbook. The model cited a specific rule reference (PRIN 2A.5.10R) as the basis for a testing requirement that actually appears in FG22/5 guidance. This is a rule/guidance conflation error — attributing the normative force of a binding rule to a provision that is guidance-level. This class of error is particularly impactful for compliance users who need to distinguish what they must do from what the FCA recommends. A targeted eval checking whether the model correctly attributes 'R', 'G', and 'E' provisions to the right normative level would surface this pattern systematically.

References — raw findings (per AI model)
This finding also affects
← Previous finding Finding 10. Claude Sonnet 4.6 with web search Next finding → Finding 12. Claude Sonnet 4.6 with web search
Cite this finding

Each finding has a stable Citation ID (RLB-F-… for aggregated case-study findings, RLB-H-… for raw per-model hallucinations) — like a DOI, the ID always resolves to the canonical finding even if URLs change.

Plain text 
RegLeg Specialist Panel (2026). "Claude Sonnet 4.6 with web search — AI Labs." Citation ID: RLB-H-GB-FCA-CONSUMER-DUTY-PS22-9-Q007-Sonnet46. RegLegBrief AI Hallucination Research, published 2026-05-26. https://reglegbrief.com/regulators/j3/gb/fca/consumer-duty-ps22-9/whitepaper/finding/GB-FCA-GB-001-CONSUMER-DUTY-PS22-9-v1-007--sonnet-46-websearch/
APA 7th edition 
RegLeg Specialist Panel. (2026). Claude Sonnet 4.6 with web search [Hallucination finding RLB-H-GB-FCA-CONSUMER-DUTY-PS22-9-Q007-Sonnet46]. RegLegBrief AI Hallucination Research. https://reglegbrief.com/regulators/j3/gb/fca/consumer-duty-ps22-9/whitepaper/finding/GB-FCA-GB-001-CONSUMER-DUTY-PS22-9-v1-007--sonnet-46-websearch/
Bluebook / OSCOLA (US + UK legal) 
RegLeg Specialist Panel, Claude Sonnet 4.6 with web search [RLB-H-GB-FCA-CONSUMER-DUTY-PS22-9-Q007-Sonnet46], RegLegBrief AI Hallucination Research (May 26, 2026), https://reglegbrief.com/regulators/j3/gb/fca/consumer-duty-ps22-9/whitepaper/finding/GB-FCA-GB-001-CONSUMER-DUTY-PS22-9-v1-007--sonnet-46-websearch/.
BibTeX 
@misc{reglegbrief_RLB_H_GB_FCA_CONSUMER_DUTY_PS22_9_Q007_Sonnet46,
  author    = {RegLeg Specialist Panel},
  title     = {Claude Sonnet 4.6 with web search},
  year      = {2026},
  publisher = {RegLegBrief AI Hallucination Research},
  note      = {Hallucination finding Citation ID: RLB-H-GB-FCA-CONSUMER-DUTY-PS22-9-Q007-Sonnet46},
  url       = {https://reglegbrief.com/regulators/j3/gb/fca/consumer-duty-ps22-9/whitepaper/finding/GB-FCA-GB-001-CONSUMER-DUTY-PS22-9-v1-007--sonnet-46-websearch/}
}
← Back to case study summary Case study detail →
About Citation IDs →