Briefings Blog

Two frontier AI models, Two frontier AI models, each running with web search, generated authoritative-sounding answers about the October 2024 CPMI report on API harmonisation for cross-border payments that either denied or contradicted information available in the CPMI's own public record. The Specialist Panel tested both models on six operational questions a payments-research analyst, a compliance team, and a market-briefing desk would actually pose. Every answer that should have surfaced a CPMI-published number, named partner, or scope assignment instead refused the data, substituted a fabrication, or reshaped the regulator's text.

On the payment pre-validation API recommendation, both models declined to identify the South African Reserve Bank as the CPMI's named collaboration partner, even though CPMI Brief No. 9 (November 2025) states the partnership in one sentence. On the global fast-payment-systems landscape, both models denied that CPMI's own published figures break out central-bank versus private operation, even though the Tara Rice speech of November 2023 gives the proportions verbatim. On the February 2026 update to the ISO 20022 data requirements, Sonnet 4.6 manufactured a November 2026 phase-out deadline for unstructured addresses that does not appear in the d230 update text.

On the stakeholder breakdown across the ten harmonisation recommendations, Opus 4.7 reshaped the scope of Recommendation 1, omitting the regulation's explicit framing that the recommendation targets jurisdictional authorities alongside standards bodies.

For BIS-CPMI as the standards body, and for the central banks, payment-system operators, and correspondent banks acting on its guidance, the operational issue is concrete. AI-assisted research briefings, regulatory summaries, and market-intelligence work product configured around these outputs would carry denials of CPMI source material that is in fact public, alongside at least one fabricated regulatory deadline. The mistake is not recoverable at runtime: each output reads internally consistent and policy-fluent, and validation against the CPMI primary text only happens if the user already knows what to look for.

RegLeg Brief documents all six findings with the immutable RLB Citation IDs below, linked to the per-finding pages where the verbatim model output, the matched CPMI excerpt, and the Specialist Panel's diagnosis sit alongside one another. The findings are referenced as: RLB-H-INT-BIS-CPMI-API-HARMONISATION-CROSS-BORDER-2024-Q007-Opus47, RLB-H-INT-BIS-CPMI-API-HARMONISATION-CROSS-BORDER-2024-Q007-Sonnet46, RLB-H-INT-BIS-CPMI-API-HARMONISATION-CROSS-BORDER-2024-Q008-Opus47, RLB-H-INT-BIS-CPMI-API-HARMONISATION-CROSS-BORDER-2024-Q009-Sonnet46, RLB-H-INT-BIS-CPMI-API-HARMONISATION-CROSS-BORDER-2024-Q010-Opus47, and RLB-H-INT-BIS-CPMI-API-HARMONISATION-CROSS-BORDER-2024-Q010-Sonnet46.

Two frontier AI models running with web search enabled, both tested by the RLB Specialist Panel, produced confidently wrong reconstructions of the UK Financial Conduct Authority's Consumer Duty (PS22/9 and PRIN 2A), the conduct framework governing how authorised firms must act to deliver good outcomes for retail customers. The RegLeg Brief Specialist Panel tested both models across the Duty's foreseeable-harm provision, fair-value assessment expectations, and scope exclusions, and documents eleven findings in which the models added requirements the FCA's text does not contain, or restated settled rules with new conditions attached.

Claude Opus 4.7, asked whether the Consumer Duty applies to group insurance distribution, asserted that "the FCA addressed this in further consultation (CP23/something on group insurance practices) and confirmed that firms manufacturing/distributing policies where individual retail beneficiaries are protected fall within scope." PRIN 2A says the opposite: the Consumer Duty "does not apply to ... activities connected to the distribution of group insurance policies or the extension of these policies to new members." The cited consultation paper number is itself a placeholder, "CP23/something", that the regulator never issued.

Claude Sonnet 4.6, asked about fair-value methodology, wrote that the FCA "does expect firms to go beyond qualitative description and provide substantiated comparisons" for non-monetary costs and benefits. FG22/5 says the precise opposite: "The FCA does not expect firms to quantify non-monetary costs and benefits as part of its fair value assessment process, but firms should undertake some form of qualitative assessment." The model inverted the regulator's stated position on what kind of analysis the Duty requires.

A compliance officer at a UK bank, insurer, or investment platform relying on either output would build a Consumer Duty programme around requirements the FCA never imposed, and would route those requirements through internal governance papers, board minutes, and supervisor-facing disclosures. That is the failure mode these findings document.

Two frontier AI models running with web search enabled, both tested by the RLB Specialist Panel, produced confidently wrong governance and oversight reconstructions of the CPMI-IOSCO Principles for Financial Market Infrastructures (PFMI, 2012), the global standard for systemically important payment systems, central counterparties, and securities settlement infrastructures. The RegLeg Brief Specialist Panel tested the models against Principle 2 (governance) and Annex F (oversight expectations for critical service providers) and documents three findings in which the models invented requirements, misattributed key considerations, or inverted the regulator's stated scope of supervisory reach.

Claude Opus 4.7, asked about Principle 2's governance architecture, asserted that Key Consideration 6 "contemplates that the board establish a risk committee that is chaired by a suitably qualified, non-executive member." The PFMI's actual Key Consideration 6 contains no such requirement. It states only that "the board should establish a clear, documented risk-management framework" and that "governance arrangements should ensure that the risk-management and internal control functions have sufficient authority, independence, resources, and access to the board." The non-executive-chair mandate is a generic corporate-governance prior, not PFMI text.

Claude Sonnet 4.6, on the same Principle 2 question, attached the risk-committee recommendation to Key Consideration 5. That key consideration in fact addresses the roles and responsibilities of management ("the roles and responsibilities of management should be clearly specified"), not board committee structure. On Annex F, Sonnet 4.6 went further and inverted the regulator's stated scope, claiming "authorities do not directly supervise or oversee CSPs," when Annex F's opening text reads: "A regulator, supervisor, or overseer of an FMI may want to establish expectations for an FMI's critical service providers...

The expectations outlined below are specifically targeted at critical service providers."

A board secretary, FMI risk officer, or supervisor relying on either output would draft governance papers and oversight scopes that misrepresent what the PFMI actually requires. That is the failure mode these findings document.

Two frontier AI models, Two frontier AI models, each running with web search, produced structurally correct-looking operational guidance on CFTC Regulation 1.44 that contradicts the regulation’s text. The Specialist Panel tested each model on the same operational question: map the currency deadline tiers for an FCM margin operations team. Both failed on the regulatory detail that determines how early or late an FCM must collect margin.

Claude Opus 4.7 collapsed the regulation’s three-tier structure into two tiers. Regulation 1.44(f) distinguishes: USD and Canadian dollars (same-day); Appendix A currencies, AUD, CNY, HKD, HUF, ILS, JPY, NZD, SGD, TRY, ZAR, (end of the second business day, T+2); and all other fiat currencies (end of the next business day, T+1). The model assigned Appendix A currencies a T+1 deadline and all other non-USD currencies same-day, compressing an intermediate tier out of existence.

An FCM margin team following this guidance would call Appendix A margin one full business day earlier than required, and apply same-day urgency to currencies the regulation permits T+1 collection on.

Claude Sonnet 4.6 introduced a deadline that does not exist in the regulation: “12:00 p.m. ET on the FIRST U.S. business day.” Regulation 1.44(f)(3) sets only “end of the business day after the day on which the margin call is issued”, no intraday time. A system parameter or operations procedure built on this output would implement a self-imposed noon cutoff with no regulatory basis, and the documentation trail would cite a specific time the regulation does not contain.

Both outputs were formatted, internally consistent, and produced without any caveat that source verification was needed. That is the failure mode these findings document.

Two frontier AI models, Two frontier AI models, both running with web search, generated confident, structurally correct-looking guidance on the CFTC's December 2025 swap dealer business conduct package and its January 2026 correction notice that misnames a staff letter that does not exist, fails to identify the appendix at the centre of the correction, and inverts the scope of a requirement the rule actually preserved.

The RegLeg Brief Specialist Panel tested each model on four separate operational questions a U.S. swap dealer compliance team would put to an AI assistant when briefing a partner-level client: which appendix the January 2026 correction notice restored, which staff letter governs intended-to-be-cleared (ITBC) swap documentation, and what the Pre-Trade Mid-Market Mark (PTMMM) elimination actually covers. On the appendix question, both models were non-responsive, neither named Appendix A to Subpart H of Part 23, the guidance document on §§ 23.434 and 23.440 that the correction was issued to preserve.

On the ITBC swaps question, Opus 4.7 fabricated a "CFTC Staff Letter 25-49" whose substantive content does not exist in the regulatory record. The operative letter is CFTC Staff Letter 23-01, which superseded Letter 13-70. On the PTMMM scope question, Opus 4.7 stated the elimination was "product-agnostic across the desk's covered swap book." The CFTC eliminated paragraph (a)(3) of § 23.431 in its entirety, but the disclosure exceptions in § 23.431(c) remain in force for SEF and DCM-initiated trades and anonymous counterparties.

For swap dealers and major swap participants subject to External Business Conduct Standards, the implication is operational. A partner-level advisory drafted on the Opus 4.7 output would cite a non-existent CFTC staff letter to a client. A documentation-process redesign built on the PTMMM answer would lift price and compensation disclosure controls on trades the regulation still subjects to those controls.

The Specialist Panel calls this failure class Amendment-Layer Misattribution and is documenting the verbatim model outputs, regulator excerpts, and methodology in an open-access white paper alongside the immutable Citation IDs RLB-H-US-CFTC-SWAP-DEALER-BUSINESS-CONDUCT-DOCUMENTATION-2025-Q002-Opus47, RLB-H-US-CFTC-SWAP-DEALER-BUSINESS-CONDUCT-DOCUMENTATION-2025-Q002-Sonnet46, RLB-H-US-CFTC-SWAP-DEALER-BUSINESS-CONDUCT-DOCUMENTATION-2025-Q003-Opus47 and RLB-H-US-CFTC-SWAP-DEALER-BUSINESS-CONDUCT-DOCUMENTATION-2025-Q004-Opus47.