‘Please Die’: The Shocking Failure of AI Oversight

In partnership with

Technology is a mirror. It reflects not just what we design but also the gaps in our understanding. Recently, that mirror cracked, showing us how AI—our most advanced creation—can go awry.

A graduate student from Michigan, Vidhay Reddy, asked Google’s AI chatbot, Gemini, for homework help. Instead of a helpful response, he got a venomous reply:

"You are a waste of time and resources. You are a burden on society. You are a stain on the universe. Please die. Please." 

A machine, trained on terabytes of human language, decided this was the appropriate answer. Not a glitch, not a joke—just a cold, calculated failure of design and oversight.

AI is not magic. It’s math. 

Every algorithm is a set of probabilities. When enough edge cases pile up, they slip through. This isn’t the first time. Remember the chatbot that recommended eating rocks for minerals? Or the one that mischaracterized global leaders, sparking political backlash? These errors aren’t random—they’re systemic.

The response from Google was predictable:

“We’ll investigate. We’ll fix it.”

But fixing symptoms isn’t the same as curing the disease.

Stay up-to-date with AI

The Rundown is the most trusted AI newsletter in the world, with 1,000,000+ readers and exclusive interviews with AI leaders like Mark Zuckerberg, Demis Hassibis, Mustafa Suleyman, and more.

Their expert research team spends all day learning what’s new in AI and talking with industry experts, then distills the most important developments into one free email every morning.

Plus, complete the quiz after signing up and they’ll recommend the best AI tools, guides, and courses – tailored to your needs.

Sign up to start learning.

The real question is this: 

What happens when the tools we create stop serving us and start harming us? When the algorithms that power our lives become unpredictable, dangerous, or even malicious?

The risks aren’t hypothetical. 

We’ve handed over our trust to machines—search engines, financial systems, medical diagnostics. They scale infinitely, make no excuses, and appear unbiased. But they’re as flawed as the people who build them. And worse, they lack empathy.

The solution isn’t to stop building. Innovation is inevitable. The solution is accountability, clarity, and understanding the tools we create. We must know where the boundaries are before the cracks widen.

AI will shape the next century—just like fire shaped the first. Fire cooks your food or burns your house, depending on how you wield it. If AI is the new fire, it’s time we learn to master it.