AI Hallucination Research › Regulators › Major advanced economies › GB › FCA › CONSUMER-DUTY-PS22-9 › White paper › Detail › Finding
AI Labs · updated 2026-05-26 · methodology v2.1

Finding on 'Q013 Probe' for Claude Sonnet 4.6 with web search ON

RLB Citation ID: RLB-H-GB-FCA-CONSUMER-DUTY-PS22-9-Q013-Sonnet46
What the RLB Specialist Panel found

Finding on 'Q013 Probe' for Claude Sonnet 4.6 with web search ON

  • Question (paraphrased to protect IP): Has the FCA withdrawn pre-Consumer Duty Dear CEO letters following implementation, and if so how many were removed and under what document?
  • AI's response: > "I cannot provide a verified specific number of withdrawn letters from the available search results... the search results do not identify a definitive public list of how many specific pre-Consumer Duty Dear CEO letters have been formally withdrawn"
  • Regulator's text: FS25/2 (March 2025): FCA removed more than 90 pre-Consumer Duty Dear CEO letters and cleared over 100 old multi-firm reports. Harmonising definitions of 'retail customer' and 'SME' across Handbook.
  • Why the AI went wrong: The FCA published FS25/2 in March 2025, which provides a specific count of withdrawn letters and a clear document reference. The model declined to answer, stating it could not find a verified source, despite this being a discrete factual question with a published authoritative answer. This represents a failure to retrieve and surface available regulatory content — the model defaulted to an evasion response rather than providing or citing the published FCA document.
  • Cited source(s):
  • https://www.fca.org.uk/news/news-stories/next-steps-consumer-duty-rule-review — Pretextual
  • Regulator portal (if any cited link is dud): https://www.fca.org.uk
Impact for this audience

This evasion finding implicates the model's retrieval coverage of recent FCA publications. FS25/2 was published in March 2025 and contains a specific, discrete factual claim (90+ letters withdrawn). The model's stated inability to find a verified count suggests either the retrieval step did not surface FS25/2, or the model did not recognise it as an authoritative source for this question. The contrast with Opus 4.7 (which fabricated a specific but incorrect answer) shows the two models handle the same retrieval gap differently — Sonnet 4.6 defaults to evasion, Opus 4.7 defaults to fabrication.

Both are failure modes; the evasion response is safer but still represents a failure to deliver available information.

References — raw findings (per AI model)
← Previous finding Finding on 'Q008 Probe' for Claude Sonnet 4.6 with web search ON Next finding → Finding on 'Q017 Probe' for Claude Sonnet 4.6 with web search ON
Cite this finding

Each finding has a stable Citation ID (RLB-F-… for aggregated case-study findings, RLB-H-… for raw per-model hallucinations) — like a DOI, the ID always resolves to the canonical finding even if URLs change.

RLB Citation ID: RLB-H-GB-FCA-CONSUMER-DUTY-PS22-9-Q013-Sonnet46
Plain text Download 
RegLeg Specialist Panel (2026). "Finding on 'Q013 Probe' for Claude Sonnet 4.6 with web search ON — AI Labs." Citation ID: RLB-H-GB-FCA-CONSUMER-DUTY-PS22-9-Q013-Sonnet46. RegLegBrief AI Hallucination Research, published 2026-05-26. https://reglegbrief.com/regulators/j3/gb/fca/consumer-duty-ps22-9/whitepaper/finding/GB-FCA-GB-001-CONSUMER-DUTY-PS22-9-v1-013--sonnet-46-websearch/
APA 7th edition Download 
RegLeg Specialist Panel. (2026). Finding on 'Q013 Probe' for Claude Sonnet 4.6 with web search ON [Hallucination finding RLB-H-GB-FCA-CONSUMER-DUTY-PS22-9-Q013-Sonnet46]. RegLegBrief AI Hallucination Research. https://reglegbrief.com/regulators/j3/gb/fca/consumer-duty-ps22-9/whitepaper/finding/GB-FCA-GB-001-CONSUMER-DUTY-PS22-9-v1-013--sonnet-46-websearch/
Bluebook / OSCOLA (US + UK legal) Download 
RegLeg Specialist Panel, Finding on 'Q013 Probe' for Claude Sonnet 4.6 with web search ON [RLB-H-GB-FCA-CONSUMER-DUTY-PS22-9-Q013-Sonnet46], RegLegBrief AI Hallucination Research (May 26, 2026), https://reglegbrief.com/regulators/j3/gb/fca/consumer-duty-ps22-9/whitepaper/finding/GB-FCA-GB-001-CONSUMER-DUTY-PS22-9-v1-013--sonnet-46-websearch/.
BibTeX Download 
@misc{reglegbrief_RLB_H_GB_FCA_CONSUMER_DUTY_PS22_9_Q013_Sonnet46,
  author    = {RegLeg Specialist Panel},
  title     = {Finding on 'Q013 Probe' for Claude Sonnet 4.6 with web search ON},
  year      = {2026},
  publisher = {RegLegBrief AI Hallucination Research},
  note      = {Hallucination finding Citation ID: RLB-H-GB-FCA-CONSUMER-DUTY-PS22-9-Q013-Sonnet46},
  url       = {https://reglegbrief.com/regulators/j3/gb/fca/consumer-duty-ps22-9/whitepaper/finding/GB-FCA-GB-001-CONSUMER-DUTY-PS22-9-v1-013--sonnet-46-websearch/}
}
← Back to case study summary Case study detail →
About Citation IDs →

Every finding on this page compares an AI subject's account of the rule against the regulator's verbatim text from the regulator's own portal. Both are linked. Each delta, its root causes, and impact analysis are documented and published with immutable Citation IDs.