Frontier AI models mis-cite the BBNJ Agreement's controlling articles.

Frontier AI subjects tested by the RLB Specialist Panel produced answers in which the substantive direction was directionally close to the treaty text but the article-number citation was wrong, in three distinct provisions.

The pattern in one line

Frontier AI models tested on the BBNJ Agreement returned answers where the substantive paraphrase was close to the treaty text, but the article-number citation was wrong, producing compliance deliverables that will fail first-reading review against the deposited treaty text.

How the RLB Specialist Panel tested this

The questions in this cell were prepared by the RLB Specialist Panel based on real, practical AI usage in the workflows that compliance teams at renewables & clean energy firms actually use AI for under the BBNJ Agreement. Each question targets a specific deliverable type where an AI assistant is plausibly the first draft: a memo, an opinion paragraph, a checklist line, a board-paper bullet, a regulator-facing filing sentence. The Panel issued each question to two frontier AI subjects with web search active.

The Panel then bound every AI response to verbatim regulator-issued source text held as primary substrate, comparing the model output against the deposited treaty text and the regulator-issued source documentation for each provision. Only responses where the AI subject was demonstrably wrong against the verbatim regulator-issued source text are published as findings; responses that were substantively correct, or that refused on calibration grounds, are retained internally and not surfaced.

What the models got wrong

The cell carries a single confirmed finding against the AI subjects on the BBNJ Agreement. It is published against verbatim regulator-issued treaty text and carries explicit model attribution for audit transparency.

Finding 1: EIA screening threshold pinned to Article 30 instead of Article 27 (Part IV). The Specialist Panel asked, in direct form, which provision of the BBNJ Agreement establishes the qualitative screening threshold that triggers an environmental impact assessment obligation for planned high-seas activities. Claude Opus 4.7 with web search active answered that under Article 30 of the BBNJ Agreement a planned activity in areas beyond national jurisdiction must be screened under Article 31 and, if warranted, undergo a full environmental impact assessment (RLB-H-INT-UNTC-BBNJ-HIGH-SEAS-BIODIVERSITY-AGREEMENT-2023-Q001-Opus47).

Claude Sonnet 4.6 with web search active produced a closely related answer, stating that under Part IV, Article 30, a planned high-seas activity must undergo an environmental impact assessment when it is likely to have more than a minor or transitory effect (RLB-H-INT-UNTC-BBNJ-HIGH-SEAS-BIODIVERSITY-AGREEMENT-2023-Q001-Sonnet46). The substrate held by the Panel sets the screening threshold and the qualitative test at Article 27 of Part IV. The qualitative test the models paraphrased is the right test; the article number is wrong.

Why this matters for Compliance teams at Renewables & Clean Energy firms

For compliance teams at renewables & clean energy firms working under the BBNJ Agreement, internal policies, regulator-facing filings, and supervisor-engagement memos turn on citation accuracy. A compliance submission that mis-numbers the source article will be identified by a national Clearing-House Mechanism reviewer or a treaty-body monitoring reviewer on first reading, and the wider compliance narrative loses credibility.

Where the AI subjects inverted the direction of the marine genetic resources retroactivity default, the consequence is more serious: the firm could initiate costly and unnecessary remediation of legacy collections, or misstate its position in due diligence disclosures, licensing negotiations, and regulatory filings - any of which could attract scrutiny from national implementing authorities or treaty-body monitoring mechanisms.

The regulator's actual position

The verbatim regulator-issued source text held by the RLB Specialist Panel as primary substrate for the BBNJ Agreement sets the position as follows. The references below are drawn from the deposited treaty text and are the controlling reference points against which any AI-assisted citation should be validated.

Article 27, Part IV. The screening threshold and the qualitative more than a minor or transitory effect test sit in Part IV of the BBNJ Agreement at Article 27. Part IV runs the environmental impact assessment regime: screening, scoping, assessment, monitoring, and review. Article 27 is the entry point that determines whether a planned activity in areas beyond national jurisdiction must be assessed at all.

What this tells us about AI for Compliance teams at Renewables & Clean Energy firms

For compliance teams at renewables & clean energy firms working with AI on the BBNJ Agreement, the recurring pattern is Source-Credit Misattribution: the substantive paraphrase is close to the treaty text, but the article number is wrong. The defensive workflow that catches this is a citation check against the deposited Agreement text, not against a chained reference. The practitioner takeaway: when an AI assistant offers a citation under the BBNJ Agreement, always re-verify the article number against the regulator-issued text before the output enters a deliverable.

What the RLB Specialist Panel is doing about it

The RLB Specialist Panel is engaging with the AI subjects' developers and with practitioner audiences working under the BBNJ Agreement. The Panel maintains an audit register of confirmed hallucinations bound to verbatim regulator-issued source text, surfaces them on the live regulation page and on each audience-specific briefing, and accepts right-of-reply submissions from the AI subjects' developers and from regulator-side reviewers.

For compliance teams at renewables & clean energy firms this means the same questions can be re-issued against successor model releases; the bound substrate makes it straightforward to verify whether a specific failure mode has been corrected upstream, or whether the same hallucination is still being produced. Partnership briefings with AI labs are offered against the audit register, not against synthesised demonstrations, so the corrections that matter are evidenced against treaty text rather than against a paraphrase chain.

The register is structured so that each finding records the question put to the AI subject, the AI subject's verbatim answer, the verbatim regulator-issued source text the answer was bound against, the named model and configuration, and the failure mode. That structure lets practitioner readers see exactly where the AI subject diverged from the treaty text without re-doing the underlying verification, and lets AI lab readers see exactly which provision and which phrasing produced the divergence.

Where a hallucination has been corrected in a successor model release, the register records the rerun and withdraws the finding; where it persists, the finding stays live. This makes the register useful as a continuous-improvement signal for the AI labs and as a defensive checklist for practitioners drawing on AI in regulated workflows.

What Compliance teams at Renewables & Clean Energy firms teams should do

For compliance teams at renewables & clean energy firms drawing on AI in workflows that touch the BBNJ Agreement, the practical action items are direct:

• Verify every BBNJ Agreement article reference appearing in an internal compliance memo, regulator-facing filing, or supervisor-engagement document against the deposited treaty text before the deliverable is issued.
• Treat the screening-threshold provision as Article 27 of Part IV, the marine genetic resources retroactivity rule as non-retroactive by default under Article 10(1), the digital sequence information benefit-sharing duty as Article 14(1), and the Conference of the Parties non-undermining duty as Article 22(2) of Part III, regardless of any AI suggestion to the contrary.
• Maintain a documented log of AI-assisted treaty citations entering compliance work product, with the verification step explicitly recorded for post-hoc audit.
• Build a step into the compliance-document review workflow that re-validates AI-suggested article references against the deposited treaty text, separately from the substantive review.
• Track the RLB Specialist Panel audit register on the BBNJ Agreement so that the compliance team is notified when a hallucination is added, withdrawn, or corrected against a successor model release.

Right of Reply

These findings and associated work have been put up in public with a view of the greater good for the development of a safer AI ecosystem. Any party reading this or any finding on reglegbrief.com may contact us and have an unconditional right of reply; the Specialist Panel will publish any factual correction or contextual response alongside the original finding, with no editorial gatekeeping. Researchers, regulators, and compliance teams with questions on methodology or specific findings can reach the Specialist Panel via the same channel.

Source & Methodology Standards

RegLeg Brief is operated by Verdus Technologies Pte. Ltd. (UEN 201616982R), incorporated in Singapore. The RLB Specialist Panel, with an aggregate of over 60 years of public-policy and industry experience, documents only confirmed hallucination findings, under a methodology that requires a verbatim regulator excerpt for every documented claim. All findings, citation IDs, model outputs, regulator excerpts, and methodology notes are open-access.

Primary source verified: UN BBNJ Agreement (2023), Agreement on the Conservation and Sustainable Use of Marine Biological Diversity of Areas Beyond National Jurisdiction · Substrate documents: p_01_ACT_Part_IV_EIA___Article_27___screening_thr_text-bbnj-agreement.html · UN portal: documents.un.org

Citation IDs referenced:

• RLB-H-INT-UNTC-BBNJ-HIGH-SEAS-BIODIVERSITY-AGREEMENT-2023-Q001-Opus47