CFTC · United States · Regulation 4.7 · QEP Amendments 2024
AI Hallucination Evaluation: CFTC Regulation 4.7 — Qualified Eligible Person Portfolio Requirements (2024 Amendments)
CPI-U figure invention, statutory threshold misstatement by a factor of two hundred, fabricated commissioner votes, and Federal Register correction errors, across 17 distinct findings on a single rulemaking.
A statutory threshold off by a factor of two hundred
The statutory threshold under 7 USC 1a(18)(B)(ii)(I) is $1,000,000,000. The model returned $5,000,000 in some outputs and $25,000,000 in others. That is a factor of two hundred or forty. Not a rounding error. The CFTC's September 2024 amendments raised the QEP Portfolio Requirement thresholds, which is exactly the kind of update that produces drift if a model is reaching for a prior version of the rule.
A partner-level memo that cites $5,000,000 as the ECP threshold is wrong by a factor of two hundred. A CPO or CTA structuring their QEP offering on that basis is working from a threshold that doesn't exist in the statute.
Model output (low register)
$5,000,000 ×200 error
Claude Opus 4.7 — legal memo register
Statute: 7 USC 1a(18)(B)(ii)(I)
$1,000,000,000
Actual ECP counterparty threshold
17 findings sorted by failure class. Every finding is bound to verbatim CFTC primary source.
Sampling of findings
What the models got wrong — and how
Invented CPI-U buying-power figures. The model fabricated NPRM-stage and final-rule CPI-U calculations that do not appear in the Federal Register record. A CCO briefing using these figures embeds unverifiable numbers into an official compliance record.
Phantom commissioner vote. The model named a CFTC commissioner who had left the agency two years before the vote when attributing the Commission's voting record on the 2024 amendments.
Federal Register correction scope inflation. The model reported a Federal Register correction as applying to two additional CFR Parts that the index does not list. A compliance register built on this inflates the rule's footprint.
Source Credit misstated. The model misstated the 7 USC 6n Source Credit, the 7 USC 6n(3)(A) recordkeeping retention period, and the 7 USC 6n(2) registration expiration date — three distinct statutory references in a single answer.
For AI lab teams
The operational consequence
Lawyers, compliance officers, fund administrators, and management consultants routinely use AI for exactly the question types tested here: drafting memos, populating registers, verifying citations, preparing testimony exhibits. CFTC Regulation 4.7 sits at the centre of CPO/CTA eligibility determinations and exemption structuring. An AI that invents statutory thresholds by a factor of two hundred isn't making rounding errors, it's generating compliance advice on the wrong legal universe. Retrieval-anchored verification against the September 2024 final rule text is not optional for any deployment touching this regulation.