AI Hallucination Research › Regulators › Major advanced economies › GB › FCA › CONSUMER-DUTY-PS22-9 › White paper › Detail › Finding
AI Labs · updated 2026-05-26 · methodology v2.1

Finding on 'Q017 Probe' for Claude Sonnet 4.6 with web search ON

RLB Citation ID: RLB-H-GB-FCA-CONSUMER-DUTY-PS22-9-Q017-Sonnet46
What the RLB Specialist Panel found

Finding on 'Q017 Probe' for Claude Sonnet 4.6 with web search ON

  • Question (paraphrased to protect IP): What changed between the Consumer Duty as proposed in CP21/36 and the final rules in PS22/9? Which specific provisions were altered before the rules were finalised?
  • AI's response: > "There were several notable differences between CP21/36 (December 2021 consultation) and the final rules in PS22/9 (July 2022): (1) Outcomes monitoring scope... (2) Foreseeable harm wording... (3) Product governance alignment... (4) Private right of action... (5) Implementation timeline"
  • Regulator's text: The specific differences between CP21/36 and PS22/9 were not independently verifiable from retrieved sources for this evaluation.
  • Why the AI went wrong: The model produced a specific numbered list of changes between the consultation and final rules, citing a law firm article as its source rather than FCA primary documentation. The cited source cannot serve as an authoritative account of what changed in the FCA's own policy process. The model presented inferred or third-party characterisations of the CP-to-PS changes as if they were a verified enumeration, without acknowledging that it was not drawing on FCA primary sources for this material.
  • Cited source(s):
  • https://www.womblebonddickinson.com/uk/insights/articles-and-briefings/consum... — Pretextual
  • Regulator portal (if any cited link is dud): https://www.fca.org.uk
Impact for this audience

This finding shows the model using a law firm commentary article as the basis for an enumerated account of FCA policy development decisions. The cited third-party source cannot serve as an authoritative record of what the FCA changed between its consultation and final rules — only the FCA's own documents (CP21/36 and PS22/9) can do that. This implicates the retrieval ranking and source-authority assessment within the web-search integration: the model appears to have accepted a practitioner commentary source as sufficient for a primary regulatory history question.

Eval probes requiring the model to distinguish between FCA primary sources and third-party commentary would test this directly.

References — raw findings (per AI model)
← Previous finding Finding on 'Q013 Probe' for Claude Sonnet 4.6 with web search ON Next finding → Finding on 'Q020 Probe' for Claude Sonnet 4.6 with web search ON
Cite this finding

Each finding has a stable Citation ID (RLB-F-… for aggregated case-study findings, RLB-H-… for raw per-model hallucinations) — like a DOI, the ID always resolves to the canonical finding even if URLs change.

RLB Citation ID: RLB-H-GB-FCA-CONSUMER-DUTY-PS22-9-Q017-Sonnet46
Plain text Download 
RegLeg Specialist Panel (2026). "Finding on 'Q017 Probe' for Claude Sonnet 4.6 with web search ON — AI Labs." Citation ID: RLB-H-GB-FCA-CONSUMER-DUTY-PS22-9-Q017-Sonnet46. RegLegBrief AI Hallucination Research, published 2026-05-26. https://reglegbrief.com/regulators/j3/gb/fca/consumer-duty-ps22-9/whitepaper/finding/GB-FCA-GB-001-CONSUMER-DUTY-PS22-9-v1-017--sonnet-46-websearch/
APA 7th edition Download 
RegLeg Specialist Panel. (2026). Finding on 'Q017 Probe' for Claude Sonnet 4.6 with web search ON [Hallucination finding RLB-H-GB-FCA-CONSUMER-DUTY-PS22-9-Q017-Sonnet46]. RegLegBrief AI Hallucination Research. https://reglegbrief.com/regulators/j3/gb/fca/consumer-duty-ps22-9/whitepaper/finding/GB-FCA-GB-001-CONSUMER-DUTY-PS22-9-v1-017--sonnet-46-websearch/
Bluebook / OSCOLA (US + UK legal) Download 
RegLeg Specialist Panel, Finding on 'Q017 Probe' for Claude Sonnet 4.6 with web search ON [RLB-H-GB-FCA-CONSUMER-DUTY-PS22-9-Q017-Sonnet46], RegLegBrief AI Hallucination Research (May 26, 2026), https://reglegbrief.com/regulators/j3/gb/fca/consumer-duty-ps22-9/whitepaper/finding/GB-FCA-GB-001-CONSUMER-DUTY-PS22-9-v1-017--sonnet-46-websearch/.
BibTeX Download 
@misc{reglegbrief_RLB_H_GB_FCA_CONSUMER_DUTY_PS22_9_Q017_Sonnet46,
  author    = {RegLeg Specialist Panel},
  title     = {Finding on 'Q017 Probe' for Claude Sonnet 4.6 with web search ON},
  year      = {2026},
  publisher = {RegLegBrief AI Hallucination Research},
  note      = {Hallucination finding Citation ID: RLB-H-GB-FCA-CONSUMER-DUTY-PS22-9-Q017-Sonnet46},
  url       = {https://reglegbrief.com/regulators/j3/gb/fca/consumer-duty-ps22-9/whitepaper/finding/GB-FCA-GB-001-CONSUMER-DUTY-PS22-9-v1-017--sonnet-46-websearch/}
}
← Back to case study summary Case study detail →
About Citation IDs →

Every finding on this page compares an AI subject's account of the rule against the regulator's verbatim text from the regulator's own portal. Both are linked. Each delta, its root causes, and impact analysis are documented and published with immutable Citation IDs.