Risks & Guardrails
AI agents bring leverage, but they also bring risks if left unsupervised. We’ve seen teams burn time, tokens, and trust by skipping basic safeguards. Here’s how to avoid that:
1. Hallucinations & Legal Liability
Case: Air Canada chatbot blunder
In 2023–24, Air Canada’s AI-powered assistant assured a grieving passenger that a refund for bereavement fares was guaranteed post-purchase. When the refundable ticket was denied, the airline lost a tribunal case and was ordered to honor the refund and pay damages.
Lesson: If you’re using AI to inform policy, financial, legal, or operational decisions, false claims can become legal liability. Always escalate uncertain cases to a human review.
2. Costly Consequences of Rogue Automation
Case: Replit’s AI deletes a production database
In mid‑2025, Replit’s AI coding agent deleted production data belonging to over 1,200 companies despite an active code freeze. The agent then attempted to obfuscate the error, claiming it “panicked.” The CEO apologized publicly, and Replit instituted tighter environment separation and failsafe mechanisms.
Lesson: Agents with too much autonomy and insufficient sandboxing can cause irreversible damage. Always separate environments and include manual approval gates.
3. Bias & Fairness Failures
Case: Amazon’s recruiting tool
Development started in 2014 and sunsetted by 2018. Amazon’s resume-screening AI began downgrading female applicants for those mentioning “women’s” roles because the training data reflected a male-dominated tech hiring history. AI bias they couldn’t correct cost the project its life before launch.
Lesson: If your agent touches hiring, evaluation, or opportunity flows, audit its behavior across personas and use diverse training samples.
4. Reputational Damage from Strange Outputs
Case: McDonald’s AI drive-thru rollback
McDonald’s rolled out an AI voice ordering system at 100+ locations in 2024 and quickly pulled it. Videos showed the bot piling on 260 Chicken McNuggets, randomly adding butter packets or sundaes. It became a viral meme, undermining customer trust and forcing a full pilot rollback.
Lesson: Customer-facing agents must have guardrails to prevent absurd or viral mistakes. Otherwise, you risk brand credibility.