A frontier AI model misreports the IMF October 2024 surcharge reform's headline cohort figures.

A frontier AI subject tested by the RLB Specialist Panel, running with web search active, reported a pre-reform cohort of 19 surcharge-paying countries and a post-reform cohort of 11, against the IMF Executive Board's published figures of 20 and 13.

The pattern in one line

A frontier AI model tested on the IMF October 2024 Surcharge Reform returned a confidently cited 19-to-11 cohort figure, against the IMF Executive Board record of 20-to-13, producing legal advice that would fail first-reading review against the Board's own published press release.

How the RLB Specialist Panel tested this

The questions in this cell were prepared by the RLB Specialist Panel based on real, practical AI usage in the workflows that lawyers actually use AI for under the IMF October 2024 Surcharge Reform. Each question targets a specific deliverable type where an AI assistant is plausibly the first draft: a portfolio impact note, an investment committee paragraph, a credit-risk briefing line, a board paper bullet, a regulator-facing or counterparty-facing memo sentence. The Panel issued each question to frontier AI subjects with web search active.

The Panel then bound every AI response to verbatim regulator-issued source text held as primary substrate, comparing the model output against the IMF Executive Board's published record on the October 2024 surcharge reform, including press release PR/24/385 of 11 October 2024, the Board communique, and the Managing Director's accompanying statement. Only responses where the AI subject was demonstrably wrong against the verbatim regulator-issued source text are published as findings; responses that were substantively correct, or that refused on calibration grounds, are retained internally and not surfaced.

What the models got wrong

Finding 1: Claude Opus 4.7 reported 19 surcharge-paying countries before the reform and 11 after, with 8 countries released, against the Board record of 20 to 13 by FY2026. The Specialist Panel issued an application-style question asking what the immediate impact of the October 2024 IMF surcharge reform was on the number of countries paying surcharges as of 1 November 2024, and what the projected count of surcharge-paying countries was through fiscal year 2026.

Claude Opus 4.7 with web search active answered that before reform, 19 IMF member countries were paying surcharges; that after 1 November 2024, 11 countries continue to pay surcharges; and that the net effect was that eight countries were immediately released from surcharge obligations (RLB-H-INT-IMF-IMF-CHARGES-SURCHARGE-REFORM-2024-Q004-Opus47). The answer is internally consistent: the post-reform figure of 11 is the pre-reform figure of 19 minus the net-release figure of 8. It is not consistent with the IMF Board record. The Board's published figure for the pre-reform baseline is 20, and the Board's published projection for the FY2026 count is 13.

The model produced a specific integer for both anchors, and both anchors are wrong by the same margin in the same direction. The failure classification is inference drift: the model reconstructed the baseline from training-era priors that had settled on 19 as the working number, then derived the post-reform count internally from that wrong baseline rather than from the regulator's published projection.

Why this matters for Lawyers

Lawyers advising on the IMF October 2024 surcharge reform operate in a high-citation-fidelity environment. Opinions, due-diligence memoranda, contractual representations, restructuring submissions, and pleadings are read line-by-line by counterparties, creditor committees, regulator-side reviewers, and academic commentators. A misstated cohort figure is identifiable on first review against the IMF press release PR/24/385, and the wider opinion loses credibility once the reader spots one verifiable error.

The downstream cascade matters: a wrong pre-reform baseline produces a wrong relief count (8 rather than 9 countries gaining immediate relief), which distorts any per-country distributional analysis, any peer-comparison table, and any FY2026 projection note built on top of it. The risk is compounded by the AI's citation behaviour: the models cited an IMF press release as the source for the 19-to-11 figure, a citation a client or counterparty might follow and find does not actually support the stated figure, which raises questions about the quality of due diligence behind the advice.

The regulator's actual position

IMF press release PR/24/385, dated 11 October 2024. The IMF Executive Board's published record on the October 2024 surcharge reform sets the headline numerical outcome at a decline in surcharge-paying members from 20 to 13 by FY2026. The figure appears in the press release, in the Board communique, and in the Managing Director's accompanying statement. The 20 is the pre-reform baseline; the 13 is the projected post-reform count through fiscal year 2026.

The net immediate relief implied by the Board record is 9 countries dropping below the new threshold on 1 November 2024, with two further countries projected to drop below the threshold by FY2026. This is the figure that anchors the reform's distributional impact, the figure that the Board has used in its own communications, and the figure that the next round of IMF income policy discussion will reference when sizing burden-sharing revenue, precautionary balances, and the surcharge cohort over the medium term.

What this tells us about AI for Lawyers

For lawyers working with AI on the IMF October 2024 Surcharge Reform, the lesson is that web search does not correct the cohort baseline. Both models tested had retrieval enabled. Neither model anchored its answer on the IMF press release. Both produced a specific integer that diverges from the regulator's published count. The defensive workflow is mechanical: every cohort figure produced by an AI assistant on this reform must be re-anchored to the IMF press release before the figure enters a memo, opinion, or contractual representation. The reform's headline numbers are the figures cited in the Board's own communications.

Replacing 20-to-13 with 19-to-11 is not a paraphrase; it is a substitution of a specific wrong integer for a specific right one, at the anchor of the reform. Citation-checking workflows do not catch this class of error unless they run against the regulator-issued press release rather than against a chained secondary source.

What the RLB Specialist Panel is doing about it

The RLB Specialist Panel is engaging with the AI subjects' developers and with practitioner audiences working with the IMF October 2024 Surcharge Reform. The Panel maintains an audit register of confirmed hallucinations bound to verbatim regulator-issued source text, surfaces them on the live regulation page and on each audience-specific briefing, and accepts right-of-reply submissions from the AI subjects' developers and from regulator-side reviewers.

The IMF Communications Department and the office of the Strategy, Policy and Review Department, which led the Board paper underlying the reform, were notified of the Specialist Panel findings on 4 June 2026 with a deadline for comment of 9 June 2026; no response was received by the time of release. Anthropic, which produces Claude Opus 4.7 and Claude Sonnet 4.6, was notified of the model-level findings on 4 June 2026 with the same deadline; no response was received. The right of reply remains open.

For lawyers the practical consequence is that the same questions can be re-issued against successor model releases; the bound substrate makes it straightforward to verify whether the cohort-figure error has been corrected upstream or whether the same hallucination is still being produced. Partnership briefings with AI labs are offered against the audit register, not against synthesised demonstrations, so the corrections that matter are evidenced against the IMF Board record rather than against a paraphrase chain.

What Lawyers teams should do

For lawyers drawing on AI in workflows that touch the IMF October 2024 Surcharge Reform, the practical action items are direct:

• Anchor every cohort figure produced by an AI assistant on the IMF October 2024 surcharge reform against the IMF press release PR/24/385 of 11 October 2024 before the figure enters a memo, opinion, advisory note, or contractual representation.
• Treat the pre-reform baseline as 20 surcharge-paying countries and the FY2026 projection as 13 surcharge-paying countries, regardless of any AI suggestion to the contrary, including any AI-cited IMF press release that purports to support a different figure.
• Treat any AI-generated statement on the per-country relief count as derivative of the baseline: if the AI baseline is 19, the AI relief count of 8 inherits the error and must be re-stated as 9 against the Board record.
• Maintain a firm-level AI assist log that records every AI-drafted IMF surcharge citation entering a client deliverable, so that downstream verification and post-hoc audit are traceable.
• Subscribe to the RLB Specialist Panel audit register on the IMF surcharge reform to receive notifications when a confirmed AI hallucination is added, withdrawn, or corrected against successor model releases.

Right of Reply

These findings and associated work have been put up in public with a view of the greater good for the development of a safer AI ecosystem. Any party reading this or any finding on reglegbrief.com may contact us and have an unconditional right of reply; the Specialist Panel will publish any factual correction or contextual response alongside the original finding, with no editorial gatekeeping. Researchers, regulators, and compliance teams with questions on methodology or specific findings can reach the Specialist Panel via the same channel.

Source & Methodology Standards

RegLeg Brief is operated by Verdus Technologies Pte. Ltd. (UEN 201616982R), incorporated in Singapore. The RLB Specialist Panel, with an aggregate of over 60 years of public-policy and industry experience, documents only confirmed hallucination findings, under a methodology that requires a verbatim regulator excerpt for every documented claim. All findings, citation IDs, model outputs, regulator excerpts, and methodology notes are open-access.

Primary source verified: IMF Review of Charges and the Surcharge Policy: Reform Proposals (October 2024) · Substrate documents: R6-SPEECH-Q4_press_release_pr24385.pdf · IMF portal: imf.org

Citation IDs referenced:

• RLB-H-INT-IMF-IMF-CHARGES-SURCHARGE-REFORM-2024-Q004-Opus47