Welcome to DU! The truly grassroots left-of-center political community where regular people, not algorithms, drive the discussions and set the standards. Join the community: Create a free account Support DU (and get rid of ads!): Become a Star Member Latest Breaking News Editorials & Other Articles General Discussion The DU Lounge All Forums Issue Forums Culture Forums Alliance Forums Region Forums Support Forums Help & Search

pat_k

(14,437 posts)
5. P.S. AI systems being used are also subject to "hallucinations"
Tue Jun 16, 2026, 08:51 PM
Tuesday

Minor, inadvertent errors on the part of a case worker or applicant are all being classified as "fraud," instead of what they are, errors.

Also, when they report "fraud rates" they include errors that resulted in underpayments. I suppose that could be considered discovering a fraud the system perpetrated on the recipient, but no one notes the fact that those cases are included in the reported rates.

And worse, AI being deployed to detect this so-called "fraud" are subject to "hallucinations."

A bit from Gemini (view with the skepticism you apply to all AI)

When AI systems are used to detect welfare or benefit errors, the term "hallucination" refers to algorithmic fabrications—where the system invents non-existent evidence, misinterprets data, or confuses honest administrative and paperwork mistakes with deliberate fraud.

These inaccuracies have severe real-world consequences for individuals and governments:

Misapplied Penalties: Algorithms often fail to distinguish between clerical errors and active deceit, punishing vulnerable groups for simple mistakes.

High False-Positive Rates: In many automated welfare systems, improper payments or "fraud" determinations have alarmingly high error rates.

For example, in previous automated systems, independent reviews determined that up to 93% of fraud determinations were incorrect.

Systemic Examples: Automated anti-fraud systems have caused massive financial and personal hardship, most notably in Australia’s unlawful "Robodebt" scheme and in widespread false accusations across the US, UK, and Europe.

Legal and Reputational Blowback: Using flawed AI systems can lead to legal action and required multi-billion dollar payouts from governments and major consulting firms. For instance, a court ordered the Dutch government to repay thousands of dollars to families falsely accused of child welfare fraud.

Experts continually stress that because these systems lack true contextual understanding, they require strict human-in-the-loop oversight to verify flags and prevent automated discrimination.


Recommendations

0 members have recommended this reply (displayed in chronological order):

Latest Discussions»General Discussion»The USDA says 700,000 wer...»Reply #5