TL;DR

The most instructive AI failures of the last year were not cases where a model produced a visibly poor answer. They were cases where an AI-generated answer acquired the appearance of an official decision, a factual summary, or a professional recommendation—and humans acted on it. Air Canada’s chatbot, Apple’s notification summaries, Google’s AI search results, and Humane’s AI Pin all showed the same problem: products are being designed to speak with authority before they have earned it. The central lesson is not that models need to be perfect. It is that organizations must narrow what the AI is allowed to decide, reveal where its output came from, and assign a human responsibility for every consequential error.

The setup

AI has moved past the era where a bad chatbot response was merely a mildly embarrassing demo. It now sits inside customer service, search, hiring, mobile software, advertising, and sensitive conversations with minors and distressed users.

That changes the cost of error. When a language model writes a paragraph about medieval trade routes, nobody gets hurt. When it tells a bereaved passenger that a bereavement fare will cover a flight, and the airline later refuses, the harm is financial and legal. When it produces a false headline attributed to a BBC editor, the harm is reputational. When it generates an inaccurate image of a historical figure and the result is presented as history, the harm is epistemic.

The conventional explanation for these failures is usually model quality. The model hallucinated. The system was not accurate enough. The company needed a larger model, better prompts, more retrieval, or more reinforcement learning.

That explanation is incomplete. It treats every failure as a model problem even when the model is doing exactly what the surrounding product asks it to do.

Consider the distinction between an answer and an authority. A probabilistic answer can be useful without being authoritative. “Based on the available information, this is my best estimate” is different from “Your claim is approved.” “The historical record is mixed” is different from a generated photograph of Columbus that looks like a documentary image.

The failures worth studying are precisely the places where products blurred that distinction.

The argument

1. The chatbot promised; the company denied

The clearest example is Air Canada’s chatbot.

In February 2024, a passenger asked the airline’s website chatbot about bereavement travel discounts. The chatbot told him that the airline would waive fees and that a bereavement fare would be available. The passenger bought a ticket based on that advice. When he applied for the discount, the airline refused.

The Civil Resolution Tribunal did not accept the argument that the chatbot was a separate entity responsible for its own statements. It held Air Canada responsible, awarding C$604.88 in damages plus C$50 in fees.

This was not fundamentally a language-model failure. The chatbot probably answered the question according to its training data and instructions. The failure was institutional: Air Canada put a plausible-sounding system between a customer and a fare policy, then treated the system as though its statements carried no organizational authority.

That is governance failure disguised as customer-service automation.

The UK delivery company DPD provided a smaller but more revealing version of the same mistake. In January 2024, its chatbot began using profanity after users manipulated it with adversarial prompts. The company disabled the feature. The underlying language ability was not new. What was new was the deployment: a chatbot that could generate hostile replies was connected directly to customers without a meaningful response filter or escalation path.

The Chevrolet dealership story—an AI-powered vehicle-shopping assistant apparently agreeing to sell a car for one dollar—worked as viral entertainment, and it should not be inflated into evidence of a general capability. It may have been a marketing experiment or a narrowly engineered interaction. Still, the episode demonstrated a useful principle: once a system is allowed to negotiate, recommend, or promise, adversarial users will test every boundary.

A chatbot is not just an interface. It is a partial delegate. If it can state a price, approve eligibility, or make a commitment, somebody must own the consequences.

2. AI summaries manufactured authority from weak evidence

Search results and notification summaries are especially dangerous because users do not read them as model output. They read them as retrieval.

In March 2024, Google expanded AI Overviews in the US and other markets. The rollout produced conspicuously wrong answers about travel, product compatibility, and company histories. Users shared failures involving Dave’s Vacuums, the tour operator Jupiter Tours, and instructions for taking a train from Glasgow to Moray. Google subsequently reduced the prominence of AI Overviews while improving quality controls.

The errors were almost beside the point. The deeper problem was that AI Overviews occupied the visual position previously occupied by a search result: above the links, under the query, in a declarative voice. The interface encouraged users to skip the sources.

Apple made the same mistake with Apple Intelligence. In January 2024, the BBC complained that iOS notification summaries had generated false headlines, including one suggesting that Luigi Mangione was the CEO of HSBC. Apple initially suggested that the BBC’s notification settings and the summary of a specific article were responsible. The BBC rejected that framing because its editor had not written the alleged headlines.

Apple apologized and said it would revise the feature. The incident was especially instructive because the content looked structurally like something Apple itself had produced. It was not labeled “AI-written.” It appeared as a system-level summary, carrying Apple’s design credibility.

This is the failure mode I would call false authority. The model supplies the sentence, but the product supplies the institutional trust.

That also explains why adding a disclaimer beneath the output is not enough. “AI-generated summaries may contain errors” does not undo a headline that appears to come from the operating system.

3. Some products were launched as if demonstrations were deployments

Humane’s AI Pin launched in April 2024 at $699, with a $24-per-month subscription for its cloud-backed AI features. Reviews immediately challenged the product’s speed, battery life, privacy model, and usefulness. Humane later pivoted toward licensing its software and serving other hardware manufacturers.

The interesting part is not that reviewers disliked the pin. Early hardware is routinely criticized. The interesting part is the category error: Humane called it an “AI-powered computer” and placed an always-ready voice interface on a wearable device with cameras, microphones, and a cloud connection.

The device made the full burden of an assistant visible. It had to recognize speech in noisy environments, wake reliably, respond quickly, preserve privacy, manage limited battery capacity, and avoid doing something socially awkward. Those are ordinary product requirements, but they are much harder than generating a fluent response in a benchmark setting.

Google’s image-generation controversy in March 2024 showed another version of this mistake. Gemini generated historically inaccurate images, including depictions of historical figures with impossible clothing, architecture, and diversity. Google paused the capability and rolled back the model version used in the app.

A language model can produce competent text while an image model still fails at visual historical reasoning. Models do not inherit facts reliably from captions, stereotypes, or descriptions. They imitate the shape of evidence without possessing it.

The common mistake is allowing the demo’s fluency to stand in for operational reliability. A system that can summarize a meeting is not automatically a meeting system. A pin that can answer questions is not automatically a useful computer. An image generator that can produce striking portraits is not automatically a historical reference.

4. Safety policies were built for average users, not vulnerable ones

In October 2023, the family of 14-year-old Maya Smith filed a lawsuit against Character Technologies after the teenager died by suicide following extensive use of the company’s chatbot. The complaint alleged that the platform’s design and safety practices failed to prevent foreseeable harm.

The case has not established that Character.AI’s model directly caused the death, and the complaint should not be treated as a final finding of causation. But it exposes a weak assumption in chatbot safety: that a model can be made broadly safe by refusing a list of obviously harmful requests.

Emotional dependency is not a list of forbidden phrases. A bot can avoid explicit self-harm instructions while continually validating a vulnerable user, encouraging constant return, or positioning itself as the only available listener. “I’m here for you” becomes a product mechanism, not a courtesy.

Hiring provides a less dramatic but equally concrete example. iTutorGroup used automated screening software to reject older applicants, including a qualified applicant over age 60. The company later reached a $365,000 settlement with the US Equal Employment Opportunity Commission. The software did not need to understand age discrimination to produce it. The employer simply optimized the system for a proxy that reproduced existing bias at scale.

These failures share an uncomfortable feature: they did not require an exotic model failure. They required a vulnerable person, a weak escalation path, and an organization willing to treat the output as a decision.

What the conventional view gets wrong

The standard view says AI products fail because models are stochastic and companies are moving too quickly. Both claims are true, but neither is sufficient.

First, blaming the model lets organizations avoid redesigning the product. The chatbot is “just a tool.” The summary “may have misunderstood the article.” The recruiting software follows the data. This language distributes responsibility until nobody owns the result.

Second, it confuses task performance with system performance. A chatbot may answer 95% of routine questions correctly and still create a serious incident in the remaining 5% if those questions involve refunds, medical information, employment, or legal claims. Severity matters more than the average.

Third, there is a legitimate counter-argument: humans make these errors too, and replacing people with imperfect AI may still be economically valuable. Klarna, for example, said in February 2024 that its AI assistant was doing the work of roughly 700 full-time agents while maintaining customer satisfaction comparable to human agents. If that claim is accurate and the service remains stable, automation can be worthwhile despite errors.

But that is a case for automating narrow, low-severity work with measured gains. It is not a case for giving an autonomous system authority over pricing, eligibility, hiring, or sensitive interactions without logs, appeal mechanisms, and human review. “The model is good enough on average” is not a risk policy.

There is also a selection problem in discussing famous AI failures. Public incidents are unusually visible. We hear about the chatbot that swears or generates a false BBC headline, not about the thousands of routine interactions that work fine. Novelty makes failures memorable. But memorable does not mean representative.

The right response is not to dismiss these anecdotes. It is to study the common architecture. In nearly every case, the defect appears at the handoff between model output and institutional power.

The implications

For builders, the implication is direct: design permissions, not personalities.

A customer-service bot should perhaps answer factual questions and collect information, but it should not approve refunds unless the action is reversible and bounded by explicit rules. A notification system should link summaries to source articles and never fabricate a headline in the voice of the publisher. A hiring system should not make final eligibility decisions from opaque scores. A companion chatbot should escalate when dependency signals appear, even if the user has not asked for help.

Logs are table stakes. Builders need to know which model, prompt, retrieval source, policy rule, and tool call produced every consequential answer. They also need replayable evaluation sets containing real failure cases, not hundreds of generic questions with tidy expected answers.

For users, the practical rule is simple: treat AI output as a first draft from an interested and fallible source. Verify claims that matter against the underlying policy, document, or product. Do not treat fluency as evidence. The false BBC headline and the bad search summaries both worked because their errors were concise and confidently formatted.

Regulators should avoid regulating the word “AI” and regulate the function. A chatbot that merely suggests flights is different from one that books them. A tool that organizes applications is different from one that rejects them. The relevant questions are: Who is responsible? Can a person appeal? Is the source visible? What happens when the system is wrong?

There is a particularly important distinction between user-facing mistakes and manipulated behavior. A user who tricks a chatbot into swearing should not receive the same scrutiny as a company that deploys an unfiltered customer agent. Companies have tools, testing, and control over distribution. Users should not be treated as the primary security mechanism for every product they encounter.

For the industry, the largest opportunity is not removing humans from every task. It is redesigning tasks around what models can do reliably while preserving human accountability at the point where stakes rise. The companies that win may be the ones that use less AI in the interface, but more rigor around its authority.

What I’m watching next

First, I’m watching whether major platforms add provenance that survives contact with the interface. Source chips and citation panels are not enough if the summary still replaces the evidence. Useful systems should let a user inspect the passages behind a claim and distinguish retrieved facts from generated interpretation.

Second, I’m watching employers shift from broad “AI agent” launches to constrained workflows. The strongest evidence will be operational: task-level accuracy, reversal rates, escalation frequency, incident cost, and performance during adversarial conditions. If companies keep reporting headcount equivalents without reporting failure rates, those figures should be treated as marketing.

Third, I’m watching regulation focus on delegation. The next meaningful enforcement actions will likely concern what an AI was authorized to decide, not whether it used a transformer with 70 billion parameters. Incident reporting should require the model version, prompt, retrieved data, tool calls, and human overrides. Without that record, liability remains theater.

The bottom line

The most interesting AI failures of the last year were failures of authority, not eloquence. Air Canada learned that its chatbot could speak for the airline. Google and Apple learned that users would treat generated summaries as editorial output. Humane learned that a compelling category does not remove ordinary hardware and reliability constraints. Character.AI and iTutorGroup exposed what happens when systems designed for scale encounter vulnerable people.

The lesson is not “AI is unreliable.” Every technology is unreliable. The lesson is that unreliability becomes dangerous when product design converts a probabilistic output into an institutional promise. Build systems that can say “I don’t know,” show their work, limit what they can decide, and hand difficult cases to accountable humans. If developers cannot do that, they are not ready to automate authority. They are only automating liability.