NatWest's AI bais paper highlights risk of hallucinations encoding bias

When you're aiming to have your AI engine figure out the relative risk of a specific transaction (for example, a housing sale), how does the description provided or generated by an LLM impact the resulting calculations?

Well, as Zachery Anderson, Chief Data & Analytics Officer at NatWest Bank highlights, it can have a significant impact - sometimes in subtle, systemic ways.

Zachery's team have written a paper on the topic.

Here's his explanation on LinkedIn:

Sometimes hallucinations aren’t just wrong — they’re bias.While testing a large language model, our team noticed something strange. It wasn’t just a made-up answer — it reflected a deeper issue. So we investigated.

What we found was concerning: hallucinations can encode bias.

In our case, the LLM invented a plausible-sounding customer persona — and layered in harmful, stereotypical assumptions. These aren’t just edge cases; they’re reminders that bias in AI can manifest in subtle, systemic ways.

We’ve published a paper on what we found and what it means for the use of generative AI in high-stakes environments like banking. I’m proud of the team for not ignoring the anomaly — but chasing it down. Raad Khraishi Devesh Batra, PhD Giulio Pelosio Robert Hankache, PhD Cristovao Iglesias Greig Cowan Graham Smith Maja Pantic Karen Dewar

This is why responsible AI matters — and why guardrails aren’t optional.

If you'd like to link directly to the PDF of the paper, follow the image below:

Nice work team NatWest (and, team UCL too!)