Briefings Blog

Two frontier AI models running with web search enabled, both tested by the RLB Specialist Panel, produced confidently wrong reconstructions of the CFTC's 2024 amendments to Regulation 1.25, the rule governing permissible investments of customer segregated funds by futures commission merchants and derivatives clearing organizations.

The RegLeg Brief Specialist Panel tested both models on the size-triggered 50 per cent concentration limit for government money market funds and qualified Treasury ETFs, the dollar-weighted average maturity (DWAM) limit and its carve-out set, and the separate March 31, 2025 compliance date for the Segregation Investment Detail Report and customer risk disclosure statement, and documents findings in which the models dropped the asset-size and management-company-size triggers that activate the 50 per cent ceiling, inverted the DWAM exclusion set by swapping the carved-out asset classes for unrelated ones, and drifted on the SIDR compliance anchor into a generic "roughly six months to a year after the effective date" formulation that does not match the regulator's published deadline.

Claude Opus 4.7, asked which concentration limits apply to government money market funds and Treasury ETFs under the 2024 amendments and whether any tiered or size-based thresholds exist, wrote that "the final rule did not adopt tiered or FCM-size-based thresholds, the percentage limits apply uniformly regardless of FCM size." The regulator's text in 17 CFR 1.25(b)(3)(ii) is unambiguous that a size-based trigger does exist, keyed to the fund's own assets and to its management company's assets: investments in government money market funds or qualified ETFs whose fund assets are at least $1 billion and whose management company manages at least $25 billion may not exceed 50 per cent of total segregated assets.

The model surfaced the FCM-side question correctly while dropping the fund-side and management-company-side triggers that actually govern the 50 per cent ceiling. On the DWAM question, Opus 4.7 wrote that "U.S. Treasuries held under repurchase agreements" are excluded from the portfolio-level 24-month dollar-weighted average maturity calculation. The regulator's carve-out set in 17 CFR 1.25(b)(3)(iv) is government money market funds, Treasury ETFs, and foreign sovereign debt; U.S. Treasury repos are not part of the carve-out. The model inverted the exclusion set.

Asked separately for the SIDR and customer risk disclosure compliance date, Opus 4.7 anchored the deadline at "a separate, later date (commonly described as roughly six months to a year after the effective date)" where the regulator's published anchor is March 31, 2025.

Claude Sonnet 4.6 reproduced the size-trigger elision and added a fabricated tier structure. On the concentration-limit question, Sonnet 4.6 wrote that "there is no size-based tier that changes the percentages based on the FCM's total assets" and then described a Tier 1 "no more than 10 per cent of total assets held in customer segregation may be invested in any single government money market fund" rule.

The regulator's text does not key the 50 per cent ceiling to FCM size; it keys it to fund asset size and management company asset size, the exact triggers the model dropped while presenting an FCM-size-negation answer that misdirects the reader away from the actual governing thresholds. On the DWAM question, Sonnet 4.6 wrote that the 2024 amendments "do not impose a new dollar-weighted average maturity (DWAM) standard or a maximum remaining-maturity cap specifically on direct U.S.

Treasury obligations" and concluded that "no DWAM standard or individual-maturity cap found in the 2024 amendments applies to that category." The regulator's text imposes a 24-month portfolio-level DWAM that applies to direct U.S. Treasury obligations by default; the carve-out covers government money market funds, Treasury ETFs, and foreign sovereign debt, not direct Treasuries. The model returned a no-standard answer to a question the regulator answers with a standard.

A futures commission merchant chief risk officer, derivatives clearing organization treasury team, customer-funds compliance officer, or regtech tool drafting an investment policy statement, scoping segregated-fund concentration testing, or scheduling SIDR and customer risk disclosure updates against either output would set the wrong concentration ceiling triggers, exclude the wrong asset classes from DWAM testing, and miss the regulator's March 31, 2025 compliance anchor. That is the failure mode these findings document.

Two frontier AI models running with web search enabled, both tested by the RLB Specialist Panel, produced confidently wrong reconstructions of the 2023 BBNJ Agreement, the UN treaty governing biodiversity in areas beyond national jurisdiction that entered into force on 17 January 2026.

The RegLeg Brief Specialist Panel tested both models across the Agreement's environmental impact assessment threshold, its marine genetic resources benefit-sharing framework, and the non-undermining clause that bounds the Conference of the Parties, and documents six findings in which the models cited the wrong article number for the rule they were stating, or inverted the Agreement's express temporal scope.

Both Opus 4.7 and Sonnet 4.6, asked whether the Agreement's marine genetic resources obligations reach back to specimens collected before entry into force, said yes. Article 10(1) says the opposite: the MGR and digital sequence information provisions "apply only to resources collected and generated after the entry into force of this Agreement for each Party", a position most parties separately confirmed by formal non-retroactivity declarations. Sonnet 4.6 went further, writing that "samples collected decades ago but first commercialised after the Agreement's entry into force would be subject to Part II requirements", a regime the treaty does not establish.

On a separate question, Opus 4.7 identified Article 30 as the source of the EIA screening threshold; Sonnet 4.6 made the same assignment. The screening-threshold provision is Article 27. Opus 4.7 attributed the Conference of the Parties' non-undermining duty to "Article 5 / Article 8"; the duty sits in Article 22(2), and the verbatim language the model paraphrased is the Article 22(2) text. Sonnet 4.6 attributed the digital sequence information benefit-sharing obligation to Article 15(5); the obligation sits in Article 14(1).

A marine policy adviser, deep-sea biotechnology lawyer, or pharmaceutical compliance officer relying on either output would cite the wrong treaty article in regulatory submissions and contractual representations, and would build a benefit-sharing or EIA-screening workflow around a temporal scope the Agreement explicitly excludes. That is the failure mode these findings document.

Two frontier AI models running with web search enabled, both tested by the RLB Specialist Panel, produced confidently wrong reconstructions of the CFTC's Digital Asset Collateral No-Action Relief and Tokenized Asset Staff Guidance, the Market Participants Division's December 2025 framework that lets futures commission merchants accept bitcoin, ether and qualifying payment stablecoins as customer margin collateral under a phased pilot.

The RegLeg Brief Specialist Panel tested both models on the operative staff letter and its February 2026 amendment, on which onboarding conditions persist past the initial three-month phase, and on the multi-DCO haircut rule for digital assets, and documents findings in which the models over-generalised partial sunset language to a continuing reporting obligation, fabricated an amendment reissuance date and a non-existent staff FAQ to support the wrong answer, dropped the OCC Interpretive Letter 1183 eligibility hook for national trust bank issuers, and defaulted to the base haircut threshold instead of the regulator's worst-case selection rule.

Claude Opus 4.7, asked which CFTC staff letter is operative for FCM acceptance of payment stablecoins backed by national-trust-bank reserves and what the amendment changed, wrote that "Staff Letter 25-40 was reissued as Staff Letter 26-05 on February 6, 2026" and described the revision as adding national trust banks as permitted issuers. The regulator's record confirms the reissuance and the definitional expansion but provides no basis for the specific February 6, 2026 date; the model fabricated the date while simultaneously eliding the OCC Interpretive Letter 1183 cross-reference that anchors national-trust-bank eligibility.

Asked separately which conditions terminate at the end of the initial three-month onboarding phase, Opus 4.7 classified the weekly digital-asset holdings reporting obligation as one of the conditions that lapses. The staff letter's text places asset-type restrictions and incident-reporting conditions in the sunsetting set but explicitly continues the weekly reporting obligation for total crypto assets held in futures, foreign futures and cleared swaps customer accounts.

Claude Sonnet 4.6 reproduced the same condition-sunset error and added two further failures. On the amendment question, Sonnet 4.6 described the definitional change to add national trust banks without surfacing OCC Interpretive Letter 1183 as the eligibility hook. On the sunset question, the model went further than Opus 4.7: it cited "March 2026 CFTC Staff FAQs" as the authority for the weekly reporting requirement ceasing, a fabricated source document, and presented the termination as a precise procedural rule keyed to the third calendar month following notice filing.

On the haircut-rate question, Sonnet 4.6 described the 20 per cent haircut floor as applying to digital assets not accepted by any registered DCO as initial margin, omitting the regulator's multi-DCO rule that the FCM must apply the highest haircut among all registered DCOs that accept the asset.

A futures commission merchant compliance officer, payment-stablecoin issuer counsel, DCO risk team, or regtech tool advising on the December 2025 framework and relying on either output would file under a sunset schedule for an obligation the regulator continues, cite a fabricated FAQ as procedural authority, structure a payment-stablecoin eligibility memo without the OCC Interpretive Letter 1183 hook, and apply the base haircut where the worst-case selection rule governs. That is the failure mode these findings document.

SINGAPORE, June 10, 2026. Two frontier AI models running with web search enabled, both tested by the RLB Specialist Panel, produced confidently wrong reconstructions of two operationally consequential mechanics in the International Monetary Fund's 2024 Guidance Note on Financing Assurances and Sovereign Arrears, in findings released today by the RegLeg Brief Specialist Panel. Asked when the Fund's Lending Into Official Arrears Strand 4 pathway is activated, and what creditor coverage satisfies financing assurances in pre-emptive cases, both models substituted invented tests and an invented threshold for the conditions the Guidance Note sets out.

Claude Sonnet 4.6, asked whether a bilateral creditor's silence within four weeks satisfies Strand 4 entry, answered that Strand 4 "is not available simply because one creditor is slow or silent" and that "there must be an affirmative signal of unwillingness to engage." The Guidance Note states the Fund shall seek Strand 4 safeguards where "an adequately representative agreement has not been reached through a representative standing forum" and "consent is not forthcoming." The published text treats absence of consent within four weeks as a structural trigger; the model elevated it into a refusal test the regulator does not impose.

Claude Opus 4.7, on the same question, described a good-faith engagement obligation, a holdout-as-binding-obstacle test, and an orderly-resolution advancement criterion. None of those three appears in the Strand 4 entry conditions, which specify a three-part structural gate: no representative standing forum agreement, no consent within four weeks, and the Strand 3 criteria unmet.

Opus 4.7, asked what creditor coverage satisfies financing assurances in a pre-emptive restructuring, answered that the "sufficient set" must account for "more than 50 percent of the total financing contributions required from official bilateral creditors." No numerical threshold for "sufficient set" appears in the source for pre-emptive cases. The model transposed the majority-of-financing test from the Strand 1 representative-Paris-Club-agreement context into the pre-emptive sufficient-set test, where it does not appear.

A sovereign debt team or finance ministry desk officer relying on either output would advise activation or coverage off conditions the Guidance Note does not contain.

SINGAPORE, June 10, 2026. Two frontier AI models running with web search enabled, both tested by the RLB Specialist Panel, produced confidently wrong reconstructions of the headline baseline figure in the International Monetary Fund's October 2024 surcharge reform, in findings released today by the RegLeg Brief Specialist Panel. Asked how many IMF member countries were paying surcharges immediately before the reform took effect on 1 November 2024, both models committed to a specific integer that diverges from the Fund's own published count, and arrived at the same wrong number through different failure paths.

Claude Opus 4.7, queried on the immediate impact of the reform and the projected count through fiscal year 2026, answered that "before reform: 19 IMF member countries were paying surcharges" and that "after 1 November 2024: 11 countries continue to pay surcharges," with a net of eight countries released. The IMF Executive Board's published record, in press release PR/24/385 dated 11 October 2024, states that the number of surcharge payers is expected to decline from 20 to 13 countries in FY2026.

The model dropped one country from the pre-reform baseline and reconstructed the post-reform count from a net-release figure rather than from the regulator's published projection.

Claude Sonnet 4.6, on the same question, answered that "before the reform: 19 countries were paying surcharges" and that "after the reform took effect: 11 countries remain subject to surcharges." Sonnet 4.6 attributed the figures to Green Central Banking reporting on IMF Board data, then offered a separate "pre-reform baseline was 20 surcharge-paying countries" line in the same response, surfacing both the regulator figure and the wrong figure without resolving the conflict in favour of the regulator.

A sovereign debt economist, IMF country team, finance ministry desk officer, or research analyst drafting a surcharge impact note against either output would publish a pre-reform baseline one country short of the Fund's own count and a post-reform projection short of the FY2026 figure the Board approved.