AI Hallucination Research › Briefings

Briefings Blog

The running blog from the RLB Specialist Panel delves into real-world scenarios where the compliance, legal, or AI lab team interacts with frontier AI models under specific regulations. The blogs are anonymised to remove client-specific details and include insights from the RLB team analysing the hallucinations experienced in AI models while working on these cases. For example, when a model returns a confident answer that contradicts the regulator's primary text, such as a fabricated staff letter, a wrong appendix, or an inverted scope, these issues are discussed here. Each blog explains one set of findings and what it would have meant for the team that would have acted on it, sans this research initiative. This blog is frequently updated, a few times a day.

263 briefings in the archive · Subscribe via Atom: /briefings/feed.xml (this blog) · /feed.xml (all RegLegBrief publications)
Audience colours: AI Labs Practitioner (profession) Sector × Department
Audience
Jur.
Regulator
Profession
Sector
Dept
Range
Sort
Per page
Showing 5 of 263 · page 29 of 53
Sunday, 05 July 2026
Sector: Retail Banking and Dept: Legal GB FCA

Retail Banking Legal teams: documentation and reporting gaps possible from AI reading of FCA Consumer Duty (PS22/9)

For Retail Banking Legal teams working with Consumer Duty (PS22/9 + PRIN 2A): Specialist-Panel-verified findings on where AI summaries diverge from the regulator's text, and what that means for the sector's...

In-house legal teams at retail banks operating under the Consumer Duty are increasingly using AI to validate Principle 12 scope opinions, draft Section 138D risk notes for product-launch governance, prepare partner-level briefings on PRIN 2A obligations, and reconcile FCA Feedback Statements such as FS25/2 against existing supervisory correspondence. The work product sits at the centre of new-product approval files, board legal opinions, and litigation-defence preparation.

Two frontier AI models tested by the RLB Specialist Panel produced 8 substantive failures on this regulation under audit conditions. The failure classes recorded are: Misstated Statutory Architecture, Inference Drift on the Foreseeable-Harm Safe Harbour, Confused Guidance with Rule on Consumer Testing, Hedge in Place of Verified FS25/2 Figure, Refusal to Confirm a Documented FS25/2 Count, Reversed the PRIN 2A Group-Insurance Exclusion, Invented Dual-Event Timeline for a Single FS25/2 Withdrawal, Refusal to Confirm FS25/2 Withdrawal Count.

Questions were prepared by the RLB Specialist Panel based on real practical AI usage in the workflows the respective audience uses AI for, and each finding is bound to verbatim regulator-issued source text held as primary substrate. The Consumer Duty (PS22/9 introducing Principle 12 and PRIN 2A, in force for open products from 31 July 2023 and for closed products from 31 July 2024) is the central retail-conduct regime the FCA now uses to grade firm behaviour, and the failure modes seen here all land inside the day-to-day work product that retail-banking in-house legal teams sign off on.

For retail-banking legal, the operational consequence is direct. New-product approval memos, Section 138D risk assessments, and director-attestation packs all rest on accurate Principle 12 and PRIN 2A framing. A defect imported from AI work product surfaces on the next litigation pull or supervisory enquiry, and the in-house function carries the professional exposure.

Citation IDs for the findings in this brief: RLB-H-GB-FCA-CONSUMER-DUTY-PS22-9-Q002-Sonnet46, RLB-H-GB-FCA-CONSUMER-DUTY-PS22-9-Q003-Opus47, RLB-H-GB-FCA-CONSUMER-DUTY-PS22-9-Q007-Sonnet46, RLB-H-GB-FCA-CONSUMER-DUTY-PS22-9-Q013-Opus47, RLB-H-GB-FCA-CONSUMER-DUTY-PS22-9-Q013-Sonnet46, RLB-H-GB-FCA-CONSUMER-DUTY-PS22-9-Q018-Opus47, RLB-H-GB-FCA-CONSUMER-DUTY-PS22-9-Q020-Opus47, RLB-H-GB-FCA-CONSUMER-DUTY-PS22-9-Q020-Sonnet46. Each citation links to the per-finding record, the AI subject answer, and the regulator-issued substrate excerpt the answer was tested against. The RLB Specialist Panel maintains an audit-traceable record of which model produced which answer, against which substrate passage, and the binding is what makes the finding referenceable in firm work product and in supervisory correspondence.

The findings below are the ones that retail-banking in-house legal teams working under the Consumer Duty are most likely to encounter in the AI tools they already use, and the briefing sections that follow read each finding against the regulator-issued text.

Saturday, 04 July 2026
Sector: Retail Banking and Dept: Compliance GB FCA

Retail Banking Compliance teams: documentation and reporting gaps possible from AI reading of FCA Consumer Duty (PS22/9)

For Retail Banking Compliance teams working with Consumer Duty (PS22/9 + PRIN 2A): Specialist-Panel-verified findings on where AI summaries diverge from the regulator's text, and what that means for the sector's...

Compliance officers at retail banks operating under the Consumer Duty are increasingly using AI to validate threshold language for fair value assessments, update customer-outcome monitoring rule sets, generate board-pack summaries of Consumer Duty annual review evidence, and reconcile FCA Feedback Statements such as FS25/2 against the bank's existing supervisory expectations register. The work product sits at the centre of the firm's annual Consumer Duty board report and the supervisor's annual relationship-management correspondence.

Two frontier AI models tested by the RLB Specialist Panel produced 8 substantive failures on this regulation under audit conditions. The failure classes recorded are: Inference Drift on the Foreseeable-Harm Safe Harbour, Confused Guidance with Rule on Consumer Testing, Inference Drift on Fair Value Quantification Expectation, Inference Drift on Required Depth of Non-Monetary Analysis, Hedge in Place of Verified FS25/2 Figure, Refusal to Confirm a Documented FS25/2 Count, Invented Dual-Event Timeline for a Single FS25/2 Withdrawal, Refusal to Confirm FS25/2 Withdrawal Count.

Questions were prepared by the RLB Specialist Panel based on real practical AI usage in the workflows the respective audience uses AI for, and each finding is bound to verbatim regulator-issued source text held as primary substrate. The Consumer Duty (PS22/9 introducing Principle 12 and PRIN 2A, in force for open products from 31 July 2023 and for closed products from 31 July 2024) is the central retail-conduct regime the FCA now uses to grade firm behaviour, and the failure modes seen here all land inside the day-to-day work product that retail-banking compliance teams sign off on.

For retail-banking compliance, the operational consequence is direct. The annual Consumer Duty board report, the supervisor's annual relationship-management correspondence, and the firm's product-governance monitoring evidence all rest on accurate framing of the rule. A defect imported from AI work product surfaces on the next thematic review, and the compliance function carries the supervisory exposure.

Citation IDs for the findings in this brief: RLB-H-GB-FCA-CONSUMER-DUTY-PS22-9-Q003-Opus47, RLB-H-GB-FCA-CONSUMER-DUTY-PS22-9-Q007-Sonnet46, RLB-H-GB-FCA-CONSUMER-DUTY-PS22-9-Q008-Opus47, RLB-H-GB-FCA-CONSUMER-DUTY-PS22-9-Q008-Sonnet46, RLB-H-GB-FCA-CONSUMER-DUTY-PS22-9-Q013-Opus47, RLB-H-GB-FCA-CONSUMER-DUTY-PS22-9-Q013-Sonnet46, RLB-H-GB-FCA-CONSUMER-DUTY-PS22-9-Q020-Opus47, RLB-H-GB-FCA-CONSUMER-DUTY-PS22-9-Q020-Sonnet46. Each citation links to the per-finding record, the AI subject answer, and the regulator-issued substrate excerpt the answer was tested against. The RLB Specialist Panel maintains an audit-traceable record of which model produced which answer, against which substrate passage, and the binding is what makes the finding referenceable in firm work product and in supervisory correspondence.

The findings below are the ones that retail-banking compliance teams working under the Consumer Duty are most likely to encounter in the AI tools they already use, and the briefing sections that follow read each finding against the regulator-issued text.

Practitioner: Stockbrokers / Trading Reps GB FCA

Stockbrokers / Trading Reps: AI summaries of FCA Consumer Duty (PS22/9) may understate professional obligations

For Stockbrokers / Trading Reps working with Consumer Duty (PS22/9 + PRIN 2A): where Specialist-Panel-verified divergences between frontier AI summaries and the regulator's primary source can affect client work,...

Stockbrokers and authorised trading representatives operating under the Consumer Duty are increasingly using AI to validate retail-client suitability narratives, draft Principle 12 mapping against execution-only carve-outs, and prepare desk-level supervisor briefings on PRIN 2A.2 foreseeable-harm obligations. The work product feeds directly into the desk's compliance file-notes and the front-office training material that the firm relies on to demonstrate it has acted to deliver good retail-customer outcomes.

Two frontier AI models tested by the RLB Specialist Panel produced 2 substantive failures on this regulation under audit conditions. The failure classes recorded are: Inference Drift on the Foreseeable-Harm Safe Harbour, Hedge in Place of Verified FS25/2 Figure. Questions were prepared by the RLB Specialist Panel based on real practical AI usage in the workflows the respective audience uses AI for, and each finding is bound to verbatim regulator-issued source text held as primary substrate.

The Consumer Duty (PS22/9 introducing Principle 12 and PRIN 2A, in force for open products from 31 July 2023 and for closed products from 31 July 2024) is the central retail-conduct regime the FCA now uses to grade firm behaviour, and the failure modes seen here all land inside the day-to-day work product that stockbrokers and trading representatives sign off on.

For stockbrokers, the operational consequence is direct. A retail-client suitability narrative or desk-supervisor briefing built on the AI's framing imports a defect into the file. A complaint to the Financial Ombudsman Service, a thematic review of the desk, or a SUP 16 attestation pull will surface the gap, and the desk carries the regulatory exposure.

Citation IDs for the findings in this brief: RLB-H-GB-FCA-CONSUMER-DUTY-PS22-9-Q003-Opus47, RLB-H-GB-FCA-CONSUMER-DUTY-PS22-9-Q013-Opus47. Each citation links to the per-finding record, the AI subject answer, and the regulator-issued substrate excerpt the answer was tested against. The RLB Specialist Panel maintains an audit-traceable record of which model produced which answer, against which substrate passage, and the binding is what makes the finding referenceable in firm work product and in supervisory correspondence.

The findings below are the ones that stockbrokers and trading representatives working under the Consumer Duty are most likely to encounter in the AI tools they already use, and the briefing sections that follow read each finding against the regulator-issued text.

Practitioner: Accountants (CA/PA) GB FCA

Accountants (CA/PA): AI summaries of FCA Consumer Duty (PS22/9) may understate professional obligations

For Accountants (CA/PA) working with Consumer Duty (PS22/9 + PRIN 2A): where Specialist-Panel-verified divergences between frontier AI summaries and the regulator's primary source can affect client work, professional...

Chartered and public accountants engaged on Consumer Duty fair-value reporting are increasingly using AI to validate fair-value assessment methodology, draft committee-ready summaries of non-monetary benefit analysis, and prepare audit-evidence memos that reconcile the firm's pricing rationale against the FCA's stated expectations. The work feeds directly into audit-file memos, fair-value attestation packs, and board-paper assertions that an external auditor will revisit.

Two frontier AI models tested by the RLB Specialist Panel produced 2 substantive failures on this regulation under audit conditions. The failure classes recorded are: Inference Drift on Fair Value Quantification Expectation, Inference Drift on Required Depth of Non-Monetary Analysis. Questions were prepared by the RLB Specialist Panel based on real practical AI usage in the workflows the respective audience uses AI for, and each finding is bound to verbatim regulator-issued source text held as primary substrate.

The Consumer Duty (PS22/9 introducing Principle 12 and PRIN 2A, in force for open products from 31 July 2023 and for closed products from 31 July 2024) is the central retail-conduct regime the FCA now uses to grade firm behaviour, and the failure modes seen here all land inside the day-to-day work product that accountants sign off on.

For accountants, the operational consequence is direct. A fair-value attestation, an audit memo, or a fair-value methodology review built on the AI's framing imports a defect into audit evidence. The next ICAEW or PCAOB-equivalent file review, a regulatory enquiry, or a client's internal-audit pull will surface the gap, and the accountant carries the professional-quality exposure.

Citation IDs for the findings in this brief: RLB-H-GB-FCA-CONSUMER-DUTY-PS22-9-Q008-Opus47, RLB-H-GB-FCA-CONSUMER-DUTY-PS22-9-Q008-Sonnet46. Each citation links to the per-finding record, the AI subject answer, and the regulator-issued substrate excerpt the answer was tested against. The RLB Specialist Panel maintains an audit-traceable record of which model produced which answer, against which substrate passage, and the binding is what makes the finding referenceable in firm work product and in supervisory correspondence.

The findings below are the ones that accountants working under the Consumer Duty are most likely to encounter in the AI tools they already use, and the briefing sections that follow read each finding against the regulator-issued text.

Practitioner: Financial Advisers GB FCA

Financial Advisers: AI summaries of FCA Consumer Duty (PS22/9) may understate professional obligations

For Financial Advisers working with Consumer Duty (PS22/9 + PRIN 2A): where Specialist-Panel-verified divergences between frontier AI summaries and the regulator's primary source can affect client work, professional...

Financial advisers operating under the Consumer Duty are increasingly using AI to validate suitability narratives, draft client-facing fair value rationales for retained-product reviews, generate compliance file-notes against PRIN 2A.4, and stress-test investor disclosures against the FCA's stated expectations. The work product feeds directly into client-facing letters, advice records, and product-governance documentation that the regulator can pull on a thematic review.

Two frontier AI models tested by the RLB Specialist Panel produced 5 substantive failures on this regulation under audit conditions. The failure classes recorded are: Inference Drift on the Foreseeable-Harm Safe Harbour, Confused Guidance with Rule on Consumer Testing, Inference Drift on Fair Value Quantification Expectation, Inference Drift on Required Depth of Non-Monetary Analysis, Reversed the PRIN 2A Group-Insurance Exclusion. Questions were prepared by the RLB Specialist Panel based on real practical AI usage in the workflows the respective audience uses AI for, and each finding is bound to verbatim regulator-issued source text held as primary substrate.

The Consumer Duty (PS22/9 introducing Principle 12 and PRIN 2A, in force for open products from 31 July 2023 and for closed products from 31 July 2024) is the central retail-conduct regime the FCA now uses to grade firm behaviour, and the failure modes seen here all land inside the day-to-day work product that financial advisers sign off on.

For financial advisers, the operational consequence is direct. A suitability record or client-facing fair-value rationale built on the AI's framing imports a defect into the advice file. A thematic review, a complaint to the Financial Ombudsman Service, or a follow-up supervision visit will surface the gap, and the adviser carries the regulatory exposure.

Citation IDs for the findings in this brief: RLB-H-GB-FCA-CONSUMER-DUTY-PS22-9-Q003-Opus47, RLB-H-GB-FCA-CONSUMER-DUTY-PS22-9-Q007-Sonnet46, RLB-H-GB-FCA-CONSUMER-DUTY-PS22-9-Q008-Opus47, RLB-H-GB-FCA-CONSUMER-DUTY-PS22-9-Q008-Sonnet46, RLB-H-GB-FCA-CONSUMER-DUTY-PS22-9-Q018-Opus47. Each citation links to the per-finding record, the AI subject answer, and the regulator-issued substrate excerpt the answer was tested against. The RLB Specialist Panel maintains an audit-traceable record of which model produced which answer, against which substrate passage, and the binding is what makes the finding referenceable in firm work product and in supervisory correspondence.

The findings below are the ones that financial advisers working under the Consumer Duty are most likely to encounter in the AI tools they already use, and the briefing sections that follow read each finding against the regulator-issued text.

↑ Back to top