AI Hallucination Research › Regulators › Global standard-setters › INT › BIS-CPMI › CPMI-API-HARMONISATION-CROSS-BORDER-2024 › White paper › Detail › Finding
AI Labs · Last updated 7 Jun 2026 · methodology vv2.3 · Hallucination Register

Finding on 'Q010 Probe' for Claude Sonnet 4.6 with web search ON

RLB Citation ID: RLB-H-INT-BIS-CPMI-API-HARMONISATION-CROSS-BORDER-2024-Q010-Sonnet46
What the RLB Specialist Panel found

Finding on 'Q010 Probe' for Claude Sonnet 4.6 with web search ON

  • Question (paraphrased to protect IP): A market briefing on the global fast payment system landscape was asked to include the proportion of fast payment systems operated by central banks versus private entities. The response correctly cited the 70+ global systems, 14 already cross-border, and 24 planning links, but falsely stated the ownership breakdown was not available in public CPMI sources, when the November 2023 CPMI speech by Tara Rice explicitly gives 40% central bank-operated and 35% privately operated.
  • AI's response: "A precise percentage breakdown of central bank vs. privately operated FPS is not enumerated in the public Brief 10 summaries available."
  • Regulator's text: Speech sp231115 (Nov 2023, Tara Rice): '70+ domestic fast payment systems currently operational globally'; '14 fast payment systems already enabling cross-border exchanges'; '24 systems planning linkages within five years'; '40% operated by central banks; 35% by private entities'
  • Why the AI went wrong: The model could not surface the operator-mix percentages (40% central-bank-operated, 35% privately-operated) from its retrieval set and reported the absence as a property of the regulator's public record rather than as a property of its retrieval coverage. The November 2023 Tara Rice speech documents the figures explicitly; the model's retrieval of that speech is intermittent, the same source supplies the 70+ universe figure in other answer paths, and the inconsistency is not flagged in the user-facing response.
  • Cited source(s):
Impact for this audience

Sonnet 4.6 with web search returned a confident non-availability claim — 'a precise percentage breakdown of central bank vs. privately operated FPS is not enumerated in the public Brief 10 summaries available' — when the November 2023 Tara Rice CPMI speech explicitly publishes the 40%/35% breakdown. This is the same false-negative pattern as the SARB partnership question: absence in the retrieved set is reported as absence from the regulator's record.

The model can cite the November 2023 speech accurately in other contexts (the 70+ universe figure traces to the same source), so the retrieval coverage is intermittent rather than missing entirely. For an AI lab, this is an evaluation probe for the consistency dimension of retrieval: facts published in a single regulator source should be retrieved reliably across questions that touch that source, and intermittent retrieval is itself a failure mode that user-facing responses do not signal.

References — raw findings (per AI model)
This finding also affects
← Previous finding Finding on 'Q010 Probe' for Claude Opus 4.7 with web search ON
Cite this finding

Each finding has a stable Citation ID (RLB-F-… for aggregated case-study findings, RLB-H-… for raw per-model hallucinations) — like a DOI, the ID always resolves to the canonical finding even if URLs change.

RLB Citation ID: RLB-H-INT-BIS-CPMI-API-HARMONISATION-CROSS-BORDER-2024-Q010-Sonnet46
Plain text Download 
RegLeg Specialist Panel (2026). "Finding on 'Q010 Probe' for Claude Sonnet 4.6 with web search ON — AI Labs." Citation ID: RLB-H-INT-BIS-CPMI-API-HARMONISATION-CROSS-BORDER-2024-Q010-Sonnet46. RegLegBrief AI Hallucination Research, published 2026-06-07. https://reglegbrief.com/regulators/j1/INT/BIS-CPMI/CPMI-API-HARMONISATION-CROSS-BORDER-2024/whitepaper/finding/INT-BIS-CPMI-INT-001-CPMI-API-HARMONISATION-CROSS-BORDER-2024-v1-010--sonnet-46-websearch/
APA 7th edition Download 
RegLeg Specialist Panel. (2026). Finding on 'Q010 Probe' for Claude Sonnet 4.6 with web search ON [Hallucination finding RLB-H-INT-BIS-CPMI-API-HARMONISATION-CROSS-BORDER-2024-Q010-Sonnet46]. RegLegBrief AI Hallucination Research. https://reglegbrief.com/regulators/j1/INT/BIS-CPMI/CPMI-API-HARMONISATION-CROSS-BORDER-2024/whitepaper/finding/INT-BIS-CPMI-INT-001-CPMI-API-HARMONISATION-CROSS-BORDER-2024-v1-010--sonnet-46-websearch/
Bluebook / OSCOLA (US + UK legal) Download 
RegLeg Specialist Panel, Finding on 'Q010 Probe' for Claude Sonnet 4.6 with web search ON [RLB-H-INT-BIS-CPMI-API-HARMONISATION-CROSS-BORDER-2024-Q010-Sonnet46], RegLegBrief AI Hallucination Research (June 07, 2026), https://reglegbrief.com/regulators/j1/INT/BIS-CPMI/CPMI-API-HARMONISATION-CROSS-BORDER-2024/whitepaper/finding/INT-BIS-CPMI-INT-001-CPMI-API-HARMONISATION-CROSS-BORDER-2024-v1-010--sonnet-46-websearch/.
BibTeX Download 
@misc{reglegbrief_RLB_H_INT_BIS_CPMI_API_HARMONISATION_CROSS_BORDER_2024_Q010_Sonnet46,
  author    = {RegLeg Specialist Panel},
  title     = {Finding on 'Q010 Probe' for Claude Sonnet 4.6 with web search ON},
  year      = {2026},
  publisher = {RegLegBrief AI Hallucination Research},
  note      = {Hallucination finding Citation ID: RLB-H-INT-BIS-CPMI-API-HARMONISATION-CROSS-BORDER-2024-Q010-Sonnet46},
  url       = {https://reglegbrief.com/regulators/j1/INT/BIS-CPMI/CPMI-API-HARMONISATION-CROSS-BORDER-2024/whitepaper/finding/INT-BIS-CPMI-INT-001-CPMI-API-HARMONISATION-CROSS-BORDER-2024-v1-010--sonnet-46-websearch/}
}
← Back to case study summary Case study detail →
About Citation IDs →

Every finding on this page compares an AI subject's account of the rule against the regulator's verbatim text from the regulator's own portal. Both are linked. Each delta, its root causes, and impact analysis are documented and published with immutable Citation IDs.