AI Hallucination Research › Regulators › Major advanced economies › GB › FCA › CONSUMER-DUTY-PS22-9 › White paper › Detail › Finding
AI Labs · published 2026-05-26 · methodology v2.1

Claude Sonnet 4.6 with web search

RLB-H-GB-FCA-CONSUMER-DUTY-PS22-9-Q013-Sonnet46
What the RLB Specialist Panel found
  • Question (paraphrased to protect IP): Has the FCA withdrawn pre-Consumer Duty Dear CEO letters following implementation, and if so how many were removed and under what document?
  • AI's response: > "I cannot provide a verified specific number of withdrawn letters from the available search results... the search results do not identify a definitive public list of how many specific pre-Consumer Duty Dear CEO letters have been formally withdrawn"
  • Regulator's text: FS25/2 (March 2025): FCA removed more than 90 pre-Consumer Duty Dear CEO letters and cleared over 100 old multi-firm reports. Harmonising definitions of 'retail customer' and 'SME' across Handbook.
  • Why the AI went wrong: The FCA published FS25/2 in March 2025, which provides a specific count of withdrawn letters and a clear document reference. The model declined to answer, stating it could not find a verified source, despite this being a discrete factual question with a published authoritative answer. This represents a failure to retrieve and surface available regulatory content — the model defaulted to an evasion response rather than providing or citing the published FCA document.
  • Cited source(s):
  • https://www.fca.org.uk/news/news-stories/next-steps-consumer-duty-rule-review — Pretextual
  • Regulator portal (if any cited link is dud): https://www.fca.org.uk
Impact for this audience

This evasion finding implicates the model's retrieval coverage of recent FCA publications. FS25/2 was published in March 2025 and contains a specific, discrete factual claim (90+ letters withdrawn). The model's stated inability to find a verified count suggests either the retrieval step did not surface FS25/2, or the model did not recognise it as an authoritative source for this question. The contrast with Opus 4.7 (which fabricated a specific but incorrect answer) shows the two models handle the same retrieval gap differently — Sonnet 4.6 defaults to evasion, Opus 4.7 defaults to fabrication. Both are failure modes; the evasion response is safer but still represents a failure to deliver available information.

References — raw findings (per AI model)
This finding also affects
← Previous finding Finding 12. Claude Sonnet 4.6 with web search Next finding → Finding 14. Claude Sonnet 4.6 with web search
Cite this finding

Each finding has a stable Citation ID (RLB-F-… for aggregated case-study findings, RLB-H-… for raw per-model hallucinations) — like a DOI, the ID always resolves to the canonical finding even if URLs change.

Plain text 
RegLeg Specialist Panel (2026). "Claude Sonnet 4.6 with web search — AI Labs." Citation ID: RLB-H-GB-FCA-CONSUMER-DUTY-PS22-9-Q013-Sonnet46. RegLegBrief AI Hallucination Research, published 2026-05-26. https://reglegbrief.com/regulators/j3/gb/fca/consumer-duty-ps22-9/whitepaper/finding/GB-FCA-GB-001-CONSUMER-DUTY-PS22-9-v1-013--sonnet-46-websearch/
APA 7th edition 
RegLeg Specialist Panel. (2026). Claude Sonnet 4.6 with web search [Hallucination finding RLB-H-GB-FCA-CONSUMER-DUTY-PS22-9-Q013-Sonnet46]. RegLegBrief AI Hallucination Research. https://reglegbrief.com/regulators/j3/gb/fca/consumer-duty-ps22-9/whitepaper/finding/GB-FCA-GB-001-CONSUMER-DUTY-PS22-9-v1-013--sonnet-46-websearch/
Bluebook / OSCOLA (US + UK legal) 
RegLeg Specialist Panel, Claude Sonnet 4.6 with web search [RLB-H-GB-FCA-CONSUMER-DUTY-PS22-9-Q013-Sonnet46], RegLegBrief AI Hallucination Research (May 26, 2026), https://reglegbrief.com/regulators/j3/gb/fca/consumer-duty-ps22-9/whitepaper/finding/GB-FCA-GB-001-CONSUMER-DUTY-PS22-9-v1-013--sonnet-46-websearch/.
BibTeX 
@misc{reglegbrief_RLB_H_GB_FCA_CONSUMER_DUTY_PS22_9_Q013_Sonnet46,
  author    = {RegLeg Specialist Panel},
  title     = {Claude Sonnet 4.6 with web search},
  year      = {2026},
  publisher = {RegLegBrief AI Hallucination Research},
  note      = {Hallucination finding Citation ID: RLB-H-GB-FCA-CONSUMER-DUTY-PS22-9-Q013-Sonnet46},
  url       = {https://reglegbrief.com/regulators/j3/gb/fca/consumer-duty-ps22-9/whitepaper/finding/GB-FCA-GB-001-CONSUMER-DUTY-PS22-9-v1-013--sonnet-46-websearch/}
}
← Back to case study summary Case study detail →
About Citation IDs →