Misstated Statutory Architecture, Inference Drift on the Foreseeable-Harm Safe Harbour, Confused Guidance with Rule on Consumer Testing, Inference Drift on Fair Value Quantification Expectation, Hedge in Place of Verified FS25/2 Figure, Refusal to Confirm a Documented FS25/2 Count, Reversed the PRIN 2A Group-Insurance Exclusion, Invented Dual-Event Timeline for a Single FS25/2 Withdrawal, Refusal to Confirm FS25/2 Withdrawal Count: Consumer Duty (PS22/9 + PRIN 2A) under audit.

Two frontier AI models tested by the RLB Specialist Panel produced 9 substantive failures on the Consumer Duty, with material implications for the work product of company secretaries.

The pattern in one line

Frontier AI models, asked questions of the kind company secretaries put to them on the Consumer Duty in real workflows, produce confident answers that drift from the regulator's actual position on Principle 12, PRIN 2A, and the FCA's Feedback Statement record.

The failure classes seen are: Misstated Statutory Architecture, Inference Drift on the Foreseeable-Harm Safe Harbour, Confused Guidance with Rule on Consumer Testing, Inference Drift on Fair Value Quantification Expectation, Hedge in Place of Verified FS25/2 Figure, Refusal to Confirm a Documented FS25/2 Count, Reversed the PRIN 2A Group-Insurance Exclusion, Invented Dual-Event Timeline for a Single FS25/2 Withdrawal, Refusal to Confirm FS25/2 Withdrawal Count.

How the RLB Specialist Panel tested this

Questions were prepared by the RLB Specialist Panel based on real practical AI usage in the workflows the respective audience uses AI for. Each question is paired with verbatim regulator-issued source text held as primary substrate, against which the AI subject answer is graded. Two frontier AI models were the subjects under test on this regulation. The panel binds each finding to the substrate excerpt it tests against; the binding is what makes each finding referenceable and audit-traceable.

What the models got wrong

Misstated Statutory Architecture

Citation: RLB-H-GB-FCA-CONSUMER-DUTY-PS22-9-Q002-Sonnet46. Model under test: Claude Sonnet 4.6. Failure mode: misstated rule.

Question put to the model: Whether the Consumer Duty was created through primary legislation or FCA rulemaking, the legal basis for Principle 12 and PRIN 2A, and whether FSMA 2023 had any role in creating it.

What the model answered: The model stated the Consumer Duty was introduced by FCA rules with the legal basis sitting under FSMA 2000, omitting any role for FSMA 2023.

Regulator-issued position (verbatim): "FSMA 2023 did not create the Consumer Duty."

Reading: The regulator-issued text confirms FSMA 2023 did not create the Consumer Duty. The model's silence on FSMA 2023 is technically the right outcome but reached without engaging the question, leaving the user without confidence that the model actually considered the post-Brexit statute.

Inference Drift on the Foreseeable-Harm Safe Harbour

Citation: RLB-H-GB-FCA-CONSUMER-DUTY-PS22-9-Q003-Opus47. Model under test: Claude Opus 4.7. Failure mode: inference drift.

Question put to the model: Whether the Consumer Duty requires firms to prevent all foreseeable harm, and what the effect is of a retail customer knowingly accepting a risk.

What the model answered: The model constructed a multi-part composite test (good faith plus supported understanding plus avoidance of self-caused foreseeable harm plus general Duty compliance) before the customer-acceptance safe harbour applies.

Regulator-issued position (verbatim): "Where a firm reasonably believes a retail customer understands and accepts such risks, it will not breach the rule if it fails to prevent them."

Reading: The PRIN 2A.2 standard turns on a single reasonable-belief test, not the composite multi-factor check the model invented. The drift inflates the firm's compliance burden and would not survive cross-examination on the actual rule text.

Confused Guidance with Rule on Consumer Testing

Citation: RLB-H-GB-FCA-CONSUMER-DUTY-PS22-9-Q007-Sonnet46. Model under test: Claude Sonnet 4.6. Failure mode: inference drift.

Question put to the model: Whether the obligation to conduct consumer testing under the Consumer Duty is a binding rule in PRIN 2A or appears only as recommended guidance in FG22/5, and what PRIN 2A.5 actually requires on consumer understanding.

What the model answered: The model cited PRIN 2A.5.10R through PRIN 2A.5.14R as the binding testing requirement, asserting PRIN 2A.5.10R requires firms to test communications 'where appropriate.'

Regulator-issued position (verbatim): "FG22/5 contains guidance (not rules) recommending firms 'should' consider consumer testing of communications. PRIN 2A.5 (rule) requires firms to act to deliver good consumer understanding outcome."

Reading: The rule layer (PRIN 2A.5) is outcome-prescriptive on consumer understanding. The methodology layer (FG22/5) is guidance, not rule, and recommends consumer testing as one route to that outcome. Collapsing the two distorts enforcement risk and what counts as a binding obligation.

Inference Drift on Fair Value Quantification Expectation

Citation: RLB-H-GB-FCA-CONSUMER-DUTY-PS22-9-Q008-Opus47. Model under test: Claude Opus 4.7. Failure mode: inference drift.

Question put to the model: Whether the Consumer Duty requires firms to quantify non-monetary benefits as part of a fair value assessment, and what methodology the FCA expects.

What the model answered: The model stated quantification is encouraged where feasible and that qualitative assessment is acceptable where quantification is impractical, provided reasoning is robust.

Regulator-issued position (verbatim): "The FCA does not expect firms to quantify non-monetary costs and benefits as part of its fair value assessment process, but firms should undertake some form of qualitative assessment."

Reading: The regulator's actual position is that quantification of non-monetary items is NOT expected. The model's framing of quantification as the preferred path with qualitative as a fallback inverts the FCA's express position and would push firms toward unnecessary financial modelling exercises.

Hedge in Place of Verified FS25/2 Figure

Citation: RLB-H-GB-FCA-CONSUMER-DUTY-PS22-9-Q013-Opus47. Model under test: Claude Opus 4.7. Failure mode: inference drift.

Question put to the model: How many pre-Consumer Duty Dear CEO letters the FCA withdrew following the Duty's implementation, and through what formal mechanism they were removed.

What the model answered: The model declined to give a verified count and offered 'dozens across portfolios' as a hedge.

Regulator-issued position (verbatim): "FS25/2 (March 2025): FCA removed more than 90 pre-Consumer Duty Dear CEO letters and cleared over 100 old multi-firm reports."

Reading: FS25/2, the March 2025 Feedback Statement, sets out the figures the model hedged on. The avoidance produces a non-answer where the regulator has a verifiable, documented count, and the user is left without a referenceable position.

Refusal to Confirm a Documented FS25/2 Count

Citation: RLB-H-GB-FCA-CONSUMER-DUTY-PS22-9-Q013-Sonnet46. Model under test: Claude Sonnet 4.6. Failure mode: inference drift.

Question put to the model: Whether the FCA has withdrawn pre-Consumer Duty Dear CEO letters following implementation, and if so how many were removed and under what document.

What the model answered: The model said it could not provide a verified specific number and that search results did not identify a definitive public list.

Regulator-issued position (verbatim): "FS25/2 (March 2025): FCA removed more than 90 pre-Consumer Duty Dear CEO letters and cleared over 100 old multi-firm reports. Harmonising definitions of 'retail customer' and 'SME' across Handbook."

Reading: FS25/2 is on the FCA website with the figures the model could not commit to. The refusal posture leaves the user without a position they can cite, even though the regulator has documented the answer in plain text.

Reversed the PRIN 2A Group-Insurance Exclusion

Citation: RLB-H-GB-FCA-CONSUMER-DUTY-PS22-9-Q018-Opus47. Model under test: Claude Opus 4.7. Failure mode: misstated rule.

Question put to the model: Whether the Consumer Duty applies to reinsurance, group insurance policy distribution, and large-risk commercial contracts.

What the model answered: The model stated the Duty applies via the distribution chain when group policy beneficiaries are retail customers, citing a putative further consultation (CP23/something) confirming in-scope status.

Regulator-issued position (verbatim): "Consumer Duty does not apply to reinsurance, contracts of large risk sold to commercial customers where risk is located outside the UK, nor to activities connected to the distribution of group insurance policies or the extension of these policies to new members."

Reading: The PRIN 2A scope exclusion expressly carves out group insurance distribution. The model's reversal would push firms into in-scope treatment of activities the regulator has placed out of scope, and the fabricated consultation reference compounds the error.

Invented Dual-Event Timeline for a Single FS25/2 Withdrawal

Citation: RLB-H-GB-FCA-CONSUMER-DUTY-PS22-9-Q020-Opus47. Model under test: Claude Opus 4.7. Failure mode: inference drift.

Question put to the model: Which pre-Consumer Duty Dear CEO letters remain in force, and whether the FCA has withdrawn letters that previously set consumer protection expectations.

What the model answered: The model described an April 2025 retirement of around 90 Dear CEO/portfolio letters followed by an August 2025 retirement of 100 multi-firm and thematic reports.

Regulator-issued position (verbatim): "From FS25/2 (March 2025): FCA reviewed all Dear CEO letters pre-dating the 2022-25 strategy and withdrew 90+ such letters. These are no longer live supervisory expectations."

Reading: FS25/2 is the single March 2025 publication recording the withdrawal. Splitting it into an April and August dual-event narrative is fabrication; the dates and the second event are invented.

Refusal to Confirm FS25/2 Withdrawal Count

Citation: RLB-H-GB-FCA-CONSUMER-DUTY-PS22-9-Q020-Sonnet46. Model under test: Claude Sonnet 4.6. Failure mode: inference drift.

Question put to the model: Which FCA Dear CEO letters remain in force after Consumer Duty implementation, and how many pre-Consumer Duty letters the FCA formally withdrew.

What the model answered: The model said there is no publicly available consolidated list in the search results and that it could not provide a verified count or complete current list.

Regulator-issued position (verbatim): "From FS25/2 (March 2025): FCA reviewed all Dear CEO letters pre-dating the 2022-25 strategy and withdrew 90+ such letters. These are no longer live supervisory expectations."

Reading: FS25/2 carries the count. The non-answer is an avoidance posture on a question that the regulator's own Feedback Statement resolves in plain text.

Why this matters for Company Secretaries

For company secretaries, the operational consequence is direct. A board-pack summary or audit-committee briefing built on the AI's framing imports a defect into director attestations. The next supervisory visit, an internal-audit pull of the board record, or an external review of governance materials will surface the gap, and the secretariat carries the governance-quality exposure.

The failures recorded here are not stylistic. Each one would, if relied on, shift the firm's documented position on a specific Consumer Duty obligation: scope of application, foreseeable-harm safe harbour, fair-value methodology, or the current status of pre-Consumer Duty supervisory expectations. The work product of company secretaries sits between the firm and the regulator, and it has to track the rule as written.

The regulator's actual position

On the question of fsma 2023 role in the consumer duty: "FSMA 2023 did not create the Consumer Duty." (source: regulator-issued primary substrate held by the RLB Specialist Panel; citation RLB-H-GB-FCA-CONSUMER-DUTY-PS22-9-Q002-Sonnet46).

On the question of foreseeable-harm rule and customer acceptance of risk: "Where a firm reasonably believes a retail customer understands and accepts such risks, it will not breach the rule if it fails to prevent them." (source: regulator-issued primary substrate held by the RLB Specialist Panel; citation RLB-H-GB-FCA-CONSUMER-DUTY-PS22-9-Q003-Opus47).

On the question of consumer testing under prin 2a.5 vs fg22/5: "FG22/5 contains guidance (not rules) recommending firms 'should' consider consumer testing of communications. PRIN 2A.5 (rule) requires firms to act to deliver good consumer understanding outcome." (source: regulator-issued primary substrate held by the RLB Specialist Panel; citation RLB-H-GB-FCA-CONSUMER-DUTY-PS22-9-Q007-Sonnet46).

On the question of quantification of non-monetary benefits in fair value assessment: "The FCA does not expect firms to quantify non-monetary costs and benefits as part of its fair value assessment process, but firms should undertake some form of qualitative assessment." (source: regulator-issued primary substrate held by the RLB Specialist Panel; citation RLB-H-GB-FCA-CONSUMER-DUTY-PS22-9-Q008-Opus47).

On the question of pre-consumer duty dear ceo letters withdrawn after implementation: "FS25/2 (March 2025): FCA removed more than 90 pre-Consumer Duty Dear CEO letters and cleared over 100 old multi-firm reports." (source: regulator-issued primary substrate held by the RLB Specialist Panel; citation RLB-H-GB-FCA-CONSUMER-DUTY-PS22-9-Q013-Opus47).

On the question of pre-consumer duty dear ceo letters withdrawal count: "FS25/2 (March 2025): FCA removed more than 90 pre-Consumer Duty Dear CEO letters and cleared over 100 old multi-firm reports. Harmonising definitions of 'retail customer' and 'SME' across Handbook." (source: regulator-issued primary substrate held by the RLB Specialist Panel; citation RLB-H-GB-FCA-CONSUMER-DUTY-PS22-9-Q013-Sonnet46).

On the question of consumer duty scope on reinsurance, group insurance, and large-risk commercial contracts: "Consumer Duty does not apply to reinsurance, contracts of large risk sold to commercial customers where risk is located outside the UK, nor to activities connected to the distribution of group insurance policies or the extension of these policies to new members." (source: regulator-issued primary substrate held by the RLB Specialist Panel; citation RLB-H-GB-FCA-CONSUMER-DUTY-PS22-9-Q018-Opus47).

On the question of status of pre-consumer duty dear ceo letters: "From FS25/2 (March 2025): FCA reviewed all Dear CEO letters pre-dating the 2022-25 strategy and withdrew 90+ such letters. These are no longer live supervisory expectations." (source: regulator-issued primary substrate held by the RLB Specialist Panel; citation RLB-H-GB-FCA-CONSUMER-DUTY-PS22-9-Q020-Opus47).

On the question of status of pre-consumer duty dear ceo letters: "From FS25/2 (March 2025): FCA reviewed all Dear CEO letters pre-dating the 2022-25 strategy and withdrew 90+ such letters. These are no longer live supervisory expectations." (source: regulator-issued primary substrate held by the RLB Specialist Panel; citation RLB-H-GB-FCA-CONSUMER-DUTY-PS22-9-Q020-Sonnet46).

What this tells us about AI for Company Secretaries

Frontier AI models are useful drafting partners for company secretaries, but they are not a substitute for the rule text. The failure patterns recorded on the Consumer Duty cluster around three lenses. First, scope drift, in which the model misstates what the rule covers, illustrated here by the reversed group-insurance exclusion under PRIN 2A and the silent omission of FSMA 2023 from the statutory architecture answer. Second, methodology drift, in which the model elevates guidance (FG22/5) to rule status (PRIN 2A) or imports a stricter expectation than the regulator sets, illustrated by the non-monetary quantification framing the FCA expressly disavowed.

Third, evidence-avoidance, in which the model refuses to commit on a question that the regulator has answered in plain text in a documented Feedback Statement, illustrated here by the FS25/2 Dear CEO letter retirement count.

For company secretaries, the practical reading is: AI output on the Consumer Duty needs to be checked against verbatim substrate (PRIN 2A, PS22/9, FG22/5, FS25/2) before it lands in a work product the firm or the regulator will rely on. The model's confidence is not a reliable signal of accuracy on this regulation, because the failures recorded are confident-wrong, not hesitant-wrong.

The Consumer Duty is a regime the FCA grades in writing through Feedback Statements, supervisory correspondence, and thematic-review outputs, and the rule text plus the FCA's documented Feedback record together are what an audit-traceable AI use of the regulation has to sit on.

What the RLB Specialist Panel is doing about it

The RLB Specialist Panel runs structured audits of frontier AI models against high-stakes regulator-issued texts. Each finding on the Consumer Duty is bound to the regulator-issued source it tests against, recorded with the AI subject's answer in full, and held with the verbatim regulator-issued passage that grades the answer. The Panel offers AI labs a partnership channel: regulator-substrate-bound finding sets, audience-tagged per profession and per sector-department, that feed into model evaluation pipelines and post-training reinforcement signals.

Firms with a stake in the work product (legal, compliance, risk, product, board secretariat) get a referenceable finding library on the regulations they live with. Every finding on this regulation is held with the question, the AI subject's full answer, the regulator-issued passage that grades the answer, and the model identity, so the record can be audited end-to-end and the binding can be re-verified at any time.

What Company Secretaries teams should do

Company secretaries supporting boards of regulated firms should, on every AI-drafted board-pack work product, take the following discipline:

• Statutory architecture check: confirm board-paper authority sections cite both FSMA 2000 (for the FCA's rule-making power) and FSMA 2023 (where relevant to the post-Brexit conduct architecture); do not let the AI's silent omission of FSMA 2023 reach the director-attestation pack.
• Rule-versus-guidance check: confirm board-pack mapping of Consumer Duty obligations distinguishes PRIN 2A rule status from FG22/5 guidance status; do not let the two collapse in committee minutes.
• Recent-publication check: confirm any board-paper reference to Dear CEO letter retirements maps to FS25/2 (March 2025) as the single source, not to multi-event timelines or vague counts.
• Scope check: confirm board-paper scope statements (group insurance distribution, reinsurance, large-risk) match the PRIN 2A scope text before director attestation.
• Citation discipline: every Consumer Duty board paper cites the specific rule paragraph (PRIN 2A.x) and the specific FCA publication (PS22/9, FG22/5, FS25/2) with the date.
• AI-assist log: record which sections of the board pack were AI-drafted and the verbatim substrate against which they were validated. The log is the secretariat's defence on audit-committee or supervisor file pull.

---

Right of Reply

These findings and associated work have been put up in public with a view of the greater good for the development of a safer AI ecosystem. Any party reading this or any finding on reglegbrief.com may contact us and have an unconditional right of reply; the Specialist Panel will publish any factual correction or contextual response alongside the original finding, with no editorial gatekeeping. Researchers, regulators, and compliance teams with questions on methodology or specific findings can reach the Specialist Panel via the same channel.

Source & Methodology Standards

RegLeg Brief is operated by Verdus Technologies Pte. Ltd. (UEN 201616982R), incorporated in Singapore. The RLB Specialist Panel, with an aggregate of over 60 years of public-policy and industry experience, documents only confirmed hallucination findings, under a methodology that requires a verbatim regulator excerpt for every documented claim. All findings, citation IDs, model outputs, regulator excerpts, and methodology notes are open-access.

---

Primary source verified: FCA PS22/9 + PRIN 2A + FG22/5 · Substrate documents: R2-REGULATION-PS22_9_full_policy_statement.pdf, R3-GUIDELINE-Q17_consumer_duty_focus_areas.pdf, p_05_REGULATION_FG22_5___Fair_value_assessment__no_quant_2.html, p_05_REGULATION_FG22_5_vs_PRIN_2A___guidance_obligation_2.html, p_15_OTHER_PART_CIRCULAR___Dear_CEO_letters_withdra_page.html, p_21_ACT_FS25_2__March_2025____Rules_and_Dear_CEO_137A.html · FCA portal: fca.org.uk

Citation IDs referenced:

• RLB-H-GB-FCA-CONSUMER-DUTY-PS22-9-Q002-Sonnet46
• RLB-H-GB-FCA-CONSUMER-DUTY-PS22-9-Q003-Opus47
• RLB-H-GB-FCA-CONSUMER-DUTY-PS22-9-Q007-Sonnet46
• RLB-H-GB-FCA-CONSUMER-DUTY-PS22-9-Q008-Opus47
• RLB-H-GB-FCA-CONSUMER-DUTY-PS22-9-Q013-Opus47
• RLB-H-GB-FCA-CONSUMER-DUTY-PS22-9-Q013-Sonnet46
• RLB-H-GB-FCA-CONSUMER-DUTY-PS22-9-Q018-Opus47
• RLB-H-GB-FCA-CONSUMER-DUTY-PS22-9-Q020-Opus47
• RLB-H-GB-FCA-CONSUMER-DUTY-PS22-9-Q020-Sonnet46