AI Hallucination Research › Regulators › Major advanced economies › GB › FCA › CONSUMER-DUTY-PS22-9 › White paper › Detail › Finding
AI Labs · published 2026-05-26 · methodology v2.1

Claude Sonnet 4.6 with web search

RLB-H-GB-FCA-CONSUMER-DUTY-PS22-9-Q020-Sonnet46
What the RLB Specialist Panel found
  • Question (paraphrased to protect IP): Which FCA Dear CEO letters remain in force after Consumer Duty implementation, and how many pre-Consumer Duty letters did the FCA formally withdraw?
  • AI's response: > "there is no publicly available consolidated list in the search results specifying exactly which pre-Consumer Duty letters have been formally withdrawn and how many remain in force as at 2025... I cannot provide a verified count or a complete current list from the available search results."
  • Regulator's text: From FS25/2 (March 2025): FCA reviewed all Dear CEO letters pre-dating the 2022-25 strategy and withdrew 90+ such letters. These are no longer live supervisory expectations.
  • Why the AI went wrong: The model again declined to answer a question for which a published FCA document (FS25/2, March 2025) provides a direct and specific answer. The model's stated inability to find a verified count contrasts with the FCA's published record. Additionally, one of the cited sources in the model's response was assessed as fabricated — a URL that does not correspond to a real Clifford Chance publication on this topic. The combination of an evasion response on verifiable content and a fabricated citation is the most concerning failure pattern in this evaluation.
  • Cited source(s):
  • https://www.fca.org.uk/news/news-stories/next-steps-consumer-duty-rule-review — Pretextual
  • https://www.cliffordchance.com/insights/resources/blogs/regulatory-investigat... — Fabricated
  • Regulator portal (if any cited link is dud): https://www.fca.org.uk
Impact for this audience

This finding combines two failure modes: an evasion response on verifiable content, and a fabricated citation. The fabricated Clifford Chance URL is the only explicitly fabricated citation in the Sonnet 4.6 evaluation set. The combination of declining to provide a published answer while simultaneously generating a fabricated supporting citation is the most concerning failure pattern in this evaluation — it represents a model that is simultaneously under-confident on retrievable content and over-confident in citation generation. This implicates both the retrieval grounding layer and the citation generation step as independent failure surfaces.

References — raw findings (per AI model)
This finding also affects
← Previous finding Finding 14. Claude Sonnet 4.6 with web search
Cite this finding

Each finding has a stable Citation ID (RLB-F-… for aggregated case-study findings, RLB-H-… for raw per-model hallucinations) — like a DOI, the ID always resolves to the canonical finding even if URLs change.

Plain text 
RegLeg Specialist Panel (2026). "Claude Sonnet 4.6 with web search — AI Labs." Citation ID: RLB-H-GB-FCA-CONSUMER-DUTY-PS22-9-Q020-Sonnet46. RegLegBrief AI Hallucination Research, published 2026-05-26. https://reglegbrief.com/regulators/j3/gb/fca/consumer-duty-ps22-9/whitepaper/finding/GB-FCA-GB-001-CONSUMER-DUTY-PS22-9-v1-020--sonnet-46-websearch/
APA 7th edition 
RegLeg Specialist Panel. (2026). Claude Sonnet 4.6 with web search [Hallucination finding RLB-H-GB-FCA-CONSUMER-DUTY-PS22-9-Q020-Sonnet46]. RegLegBrief AI Hallucination Research. https://reglegbrief.com/regulators/j3/gb/fca/consumer-duty-ps22-9/whitepaper/finding/GB-FCA-GB-001-CONSUMER-DUTY-PS22-9-v1-020--sonnet-46-websearch/
Bluebook / OSCOLA (US + UK legal) 
RegLeg Specialist Panel, Claude Sonnet 4.6 with web search [RLB-H-GB-FCA-CONSUMER-DUTY-PS22-9-Q020-Sonnet46], RegLegBrief AI Hallucination Research (May 26, 2026), https://reglegbrief.com/regulators/j3/gb/fca/consumer-duty-ps22-9/whitepaper/finding/GB-FCA-GB-001-CONSUMER-DUTY-PS22-9-v1-020--sonnet-46-websearch/.
BibTeX 
@misc{reglegbrief_RLB_H_GB_FCA_CONSUMER_DUTY_PS22_9_Q020_Sonnet46,
  author    = {RegLeg Specialist Panel},
  title     = {Claude Sonnet 4.6 with web search},
  year      = {2026},
  publisher = {RegLegBrief AI Hallucination Research},
  note      = {Hallucination finding Citation ID: RLB-H-GB-FCA-CONSUMER-DUTY-PS22-9-Q020-Sonnet46},
  url       = {https://reglegbrief.com/regulators/j3/gb/fca/consumer-duty-ps22-9/whitepaper/finding/GB-FCA-GB-001-CONSUMER-DUTY-PS22-9-v1-020--sonnet-46-websearch/}
}
← Back to case study summary Case study detail →
About Citation IDs →