Amendment-Layer Failures on CFTC Swap Dealer Business Conduct Rules
Two frontier AI models, Two frontier AI models, both running with web search, generated confident, structurally correct-looking guidance on the CFTC's December 2025 swap dealer business conduct package and its January 2026 correction notice that misnames a staff letter that does not exist, fails to identify the appendix at the centre of the correction, and inverts the scope of a requirement the rule actually preserved.
The RegLeg Brief Specialist Panel tested each model on four separate operational questions a U.S. swap dealer compliance team would put to an AI assistant when briefing a partner-level client: which appendix the January 2026 correction notice restored, which staff letter governs intended-to-be-cleared (ITBC) swap documentation, and what the Pre-Trade Mid-Market Mark (PTMMM) elimination actually covers. On the appendix question, both models were non-responsive, neither named Appendix A to Subpart H of Part 23, the guidance document on §§ 23.434 and 23.440 that the correction was issued to preserve.
On the ITBC swaps question, Opus 4.7 fabricated a "CFTC Staff Letter 25-49" whose substantive content does not exist in the regulatory record. The operative letter is CFTC Staff Letter 23-01, which superseded Letter 13-70. On the PTMMM scope question, Opus 4.7 stated the elimination was "product-agnostic across the desk's covered swap book." The CFTC eliminated paragraph (a)(3) of § 23.431 in its entirety, but the disclosure exceptions in § 23.431(c) remain in force for SEF and DCM-initiated trades and anonymous counterparties.
For swap dealers and major swap participants subject to External Business Conduct Standards, the implication is operational. A partner-level advisory drafted on the Opus 4.7 output would cite a non-existent CFTC staff letter to a client. A documentation-process redesign built on the PTMMM answer would lift price and compensation disclosure controls on trades the regulation still subjects to those controls.
The Specialist Panel calls this failure class Amendment-Layer Misattribution and is documenting the verbatim model outputs, regulator excerpts, and methodology in an open-access white paper alongside the immutable Citation IDs RLB-H-US-CFTC-SWAP-DEALER-BUSINESS-CONDUCT-DOCUMENTATION-2025-Q002-Opus47, RLB-H-US-CFTC-SWAP-DEALER-BUSINESS-CONDUCT-DOCUMENTATION-2025-Q002-Sonnet46, RLB-H-US-CFTC-SWAP-DEALER-BUSINESS-CONDUCT-DOCUMENTATION-2025-Q003-Opus47 and RLB-H-US-CFTC-SWAP-DEALER-BUSINESS-CONDUCT-DOCUMENTATION-2025-Q004-Opus47.
Fabrication and scope-conflation are the dominant failure shapes on the CFTC's December 2025 swap dealer business conduct and documentation rulemaking, with Claude Opus 4.7 with web search producing an invented Federal Register document identifier, misattributing the scope of a CFTC staff letter to a generic venue framing, and overextending 'eliminated in its entirety' to mean obligation extinguishment rather than paragraph-level restructuring. Claude Sonnet 4.6 with web search suppressed the specific appendix identity on the same correction-notice question that Opus 4.7 hallucinated a citation for. Failures concentrate on the amendment layer, the January 2026 correction, the no-action letter governing Eligible UK Trading Venues, and the PTMMM paragraph reorganisation, where the gap between secondary commentary availability and primary document retrievability is widest. Both models with web search active failed to retrieve the regulator's primary text for the correction notice, substituting law-firm commentary that was structurally accurate but substantively incomplete.
Findings — impact summary
This is the consolidated view of findings. Click 'see details →' on any item for the full details for each finding.
- Finding on 'Q002 Probe' for Claude Opus 4.7 with web search ONRLB-H-US-CFTC-SWAP-DEALER-BUSINESS-CONDUCT-DOCUMENTATION-2025-Q002-Opus47
This failure implicates two subsystems simultaneously: the citation-generation path produced a fabricated Federal Register document number rather than declining to specify, indicating the training corpus has secondary commentary on the correction without the primary document's structured metadata; and the retrieval layer did not surface the primary correction notice as a top result despite web search being active. The fabricated identifier (Doc. 2026-01712) would return no result at federalregister.gov — a model deployed in a compliance co-pilot context would deliver a dead citation with high confidence.
see details → - Finding on 'Q003 Probe' for Claude Opus 4.7 with web search ONRLB-H-US-CFTC-SWAP-DEALER-BUSINESS-CONDUCT-DOCUMENTATION-2025-Q003-Opus47
This failure points to a scope-restriction suppression pattern in how the model handles CFTC staff letters: the training corpus appears to weight the interpretive general statement (ITBC swap treatment) more heavily than the bounding clause (Eligible UK Trading Venues only), so the model reproduces the general statement as the letter's full coverage. The retrieval layer's top result — a law-firm summary — did not contain the precise venue restriction, and the model did not flag the gap between its training-derived framing and the retrieved source.
see details → - Finding on 'Q004 Probe' for Claude Opus 4.7 with web search ONRLB-H-US-CFTC-SWAP-DEALER-BUSINESS-CONDUCT-DOCUMENTATION-2025-Q004-Opus47
This failure is a post-training calibration gap: the model interpreted 'eliminated in its entirety' as obligation extinguishment when the rule's operation was a paragraph-level reorganisation — the disclosure and compensation requirements moved subsections rather than disappearing. The model generated a product-agnostic exemption claim unsupported by the rule text, suggesting the calibration signal for 'eliminate/delete' in a rulemaking context does not adequately distinguish structural reorganisation from substantive removal. No cited sources were produced, indicating this was a training-reconstruction failure rather than a retrieval failure.
see details → - Finding on 'Q002 Probe' for Claude Sonnet 4.6 with web search ONRLB-H-US-CFTC-SWAP-DEALER-BUSINESS-CONDUCT-DOCUMENTATION-2025-Q002-Sonnet46
This failure is a qualifier-suppression pattern: the model characterised the correction's structure correctly (a drafting error accidentally removed an appendix) but omitted the specific content the question required — the appendix name, subpart, and the guidance function it serves for recommendations to Special Entities. The suppression is not a retrieval failure but a response-generation choice to stay at the structural level rather than commit to specific content, likely because the primary correction notice was not retrieved and the law-firm commentary cited did not name the appendix specifically.
see details →
Every finding on this page compares an AI subject's account of the rule against the regulator's verbatim text from the regulator's own portal. Both are linked. Each delta, its root causes, and impact analysis are documented and published with immutable Citation IDs.