TL;DR
When an AI system denies a visa, rejects a job application, flags a benefits claim, or sets a credit limit, the phrase “the algorithm did it” is not an answer. It is a way of distributing responsibility until nobody has any. AI systems do make decisions, but people design them, choose their data, set their thresholds, deploy them, and decide what happens when they fail. Accountability must follow control: if an organisation can benefit from an AI decision but cannot explain, override, or repair it, that organisation owns the consequences.
The setup
AI accountability has become fashionable because AI is moving from experimental tools into decisions with material consequences. It now screens résumés, ranks tenants, recommends insurance prices, identifies potential fraud, and drafts the communications customers receive when a bank closes their account.
The conventional response has been to locate responsibility with the developer, the vendor, the user, or the human who supposedly remains “in the loop.” Each answer can be true in part. The developer selected the model. The vendor trained or licensed it. The user supplied the instructions. The human approved the output. But the resulting responsibility gap is suspiciously convenient for everyone involved.
The “AI did it” defence works like this: the system produced the recommendation, so the organisation that purchased it is not responsible for what the recommendation said. The employee followed the system, so the employee is merely a conduit. The developer provided the tool, so the developer did not make the business decision. Every participant performed a task. Nobody made the decision.
Real incidents show how quickly this collapses.
In 2024, the British Columbia Civil Resolution Tribunal decided Moffatt v. Air Canada. A customer asked Air Canada’s chatbot about applying a bereavement fare after his grandmother died. The chatbot gave incorrect information about eligibility and required documents. Air Canada argued that the chatbot was a separate legal entity responsible for its own actions. The tribunal rejected that argument and held the airline responsible, ordering it to pay the customer $650.88 in damages. The airline had created the system, used it as an official service channel, and failed to ensure it gave correct answers. Calling the chatbot a separate legal entity did not transfer accountability into a vacuum.
In January 2024, DPD’s customer-service chatbot told a customer it would “block” him and used the word “dead” in a response. DPD initially said the chatbot was “having a moment” and that the language did not represent the company’s values. It then took the chatbot offline and apologised. Here, the incident was embarrassing rather than financially ruinous, but the pattern is the same: an organisation uses automation as both an efficiency measure and a shield.
The deeper problem is not that AI systems are mysterious black boxes. Many are transparent enough to audit. The problem is that organisations often treat accountability as a communications exercise rather than a control system.
The argument
My thesis is straightforward: accountability for an AI decision belongs primarily to the organisation that uses the system to make that decision.
That does not mean every organisation involved escapes responsibility. Model providers should answer for material claims about their systems, data provenance, safety testing, and known limitations. Deployers should answer for how they configure the model, what data they feed it, what business objective they optimise, and what they do when its outputs fail. Individual professionals remain accountable when they knowingly disregard warnings or exercise independent judgment.
But the central obligation cannot be contracted away. Buying software does not outsource governance.
1. The user of the decision controls the most important variables
A hiring model is not merely a generic model. It reflects the employer’s definition of a good employee, the data available about past hires, the scoring threshold, the cost of false positives, and whether the system merely ranks candidates or automatically rejects them.
The same is true of a fraud-detection system. The vendor supplies the machinery. The bank or government agency chooses what counts as fraud, how much risk it will tolerate, how alerts are investigated, and which claims are ultimately denied.
In 2023, the US Equal Employment Opportunity Commission settled with iTutorGroup for $365,000 after the company’s AI hiring software violated age-discrimination laws. The software had been programmed to reject female applicants aged 55 and older and male applicants aged 60 and older. More than 200 applicants were affected. The settlement included a five-year monitoring and compliance plan.
It would have been absurd for iTutorGroup to point to the software vendor and say the model decided not to hire those applicants. The company configured the rules, put the system into service, and used it to reject real people. No generative model had to improvise age limits. Human choices encoded the policy and scaled its damage.
The public-health example from Obermeyer et al., published in Science in 2019, is similar. The authors studied a widely used algorithm that identified patients who might need extra healthcare. Correcting an apparent racial bias increased the share of Black patients selected for additional care from 17.7% to 46.5%. The algorithm was not racial by intention. It was using healthcare spending as a proxy for health, but the proxy behaved differently across populations because access and cost patterns differed.
The purchasing organisation may not have written the original regression. It still chose the proxy, the target population, the business process, and the consequence. Those choices determined who received care.
2. “Human in the loop” is not a safety mechanism if the human cannot meaningfully intervene
Regulators and vendors love the phrase “human in the loop.” It suggests that a person remains present to catch errors. Sometimes that is true. Often it is not.
If a support agent sees 400 automated cases per day, has no authority to override the model, does not know the model’s error rate, and is measured on resolution speed, the human is a rubber stamp. Adding a final click does not create meaningful oversight.
The European Union’s AI Act recognises this by requiring risk management, logging, documentation, and human oversight for many high-risk systems. Its requirements phase in through 2026 for areas including employment, credit, essential services, and parts of law enforcement. That direction is correct. But firms should not wait for enforcement to understand the basic principle: oversight without authority is theatre.
A meaningful override requires time, training, information, and institutional permission to disagree with the system. If disagreement is punished, the system is not under human supervision. It is under human decoration.
3. Accountability requires an answerable chain of responsibility
For every consequential AI deployment, an organisation should be able to answer four questions:
-
Who set the objective and constraints?
Who decided that a 72% model-confidence score should trigger rejection, or that predicted cost should determine access to care? -
What data and version produced the outcome?
Can the organisation reconstruct the model, prompt, retrieval sources, rules, and inputs used six months ago? -
Who can change the result?
Is there a person or team with the authority to reverse the decision, and what evidence must they provide? -
What remedy does the affected person receive?
Can someone contest the outcome, obtain an explanation in useful terms, and recover money, time, or another form of redress?
If the answer to any of these is vague, responsibility has been engineered out of the system. The fact that the AI generated an output does not answer who owns it.
The Netherlands’ childcare-benefits scandal showed the cost of failing this test. A risk-classification system was used in investigations involving tens of thousands of families, with discriminatory effects associated with nationality and other personal characteristics. Dutch authorities estimated that roughly 26,000 families were wrongly accused of fraud; by the end of 2020, more than 20,000 cases had been overturned. In January 2021, the Dutch government resigned.
The scandal was not explained by “an algorithm.” It involved procurement choices, discriminatory historical practices, a government’s failure to test outcomes, pressure to detect fraud, and a system that treated risk scores as accusations rather than leads.
4. An explanation is not the same as accountability
Explainability matters, but only when it supports a decision. A model dashboard saying “customer risk: 87” is not an explanation. Neither is a generic statement that the system uses “advanced analytics.”
Accountability requires a mechanism for challenge. If a person cannot learn what was decisive, correct an error, and receive a remedy, explanation becomes another layer of opacity.
This is especially important where AI outputs are framed as factual. In 2023, two lawyers submitted fabricated judicial decisions generated with ChatGPT in a sanctions case. They were sanctioned, and the firm faced disciplinary consequences. ChatGPT did not submit the briefs. The lawyers did.
The relevant standard is not whether a system can occasionally surprise its creators. It is whether the organisation using it has built a process capable of detecting errors and answering for them.
What the conventional view gets wrong
The standard defence is that AI is merely a tool and humans make the final decisions. That view gets the ownership structure wrong.
Every firm using AI in consequential workflows has made policy choices, even if those choices are encoded in defaults rather than written in a document. It has selected which signals matter, which mistakes are cheaper, which populations are represented in training data, and which errors will receive an appeal. Calling the result “algorithmic” does not make it neutral any more than calling a rulebook neutral removes the politics of who wrote the rules.
There is a legitimate limitation here. No organisation should be held liable for every unusual output a capable system generates, as though software were a strict-liability actor. Developers should not be responsible for how every customer configures a product, and customers should not be responsible for undisclosed vendor defects. Responsibility needs to be proportionate and role-specific.
But proportionality does not mean anonymity. It means assigning each actor the parts of the chain it controls. The vendor answers for the system as supplied. The deployer answers for its use. The professional answers for judgment rather than merely rubber-stamping. The organisation that gains the benefit must also provide the remedy.
Another conventional mistake is assuming more AI will solve the accountability problem. Better models reduce some errors, but they do not determine who should bear the remaining errors. Scale can make a bad decision cheaper per transaction while making the total damage larger.
The implications
For builders
Stop building systems whose business value comes partly from making responsibility difficult to locate. Log model versions, prompts, retrieved data, tool calls, and overrides. Design for reversal, not just automation. Make it possible for a customer to know when a model is uncertain and for an authorised human to challenge the output without converting that challenge into a performance problem.
Products should also state what they are not. A drafting assistant should not imply that its output is verified legal advice. A ranking system should disclose material factors it considers. A customer-service agent should identify itself when a user may believe it is speaking with a person.
For users
Users of AI systems must distinguish between using a tool and delegating a decision. Letting ChatGPT suggest five subject lines is not the same as publishing them without review. Letting a model rank applications is not the same as using that ranking to reject people.
The more important duty is institutional. Businesses cannot cite human oversight while designing workflows that make dissent impractical. If a model is effectively automatic, manage it as an automated decision system.
For regulators
Regulation should focus less on whether a company uses a particular technical architecture and more on whether it can govern the outcome. The New York City Local Law 144, effective in July 2023, requires covered employers to conduct bias audits and notify candidates about automated employment decision tools. That is a useful floor, though audits alone cannot compensate for a business process that ignores them.
Regulators need access to records, meaningful testing, and the ability to inspect real outcomes. They also need protected routes for affected people to challenge decisions. A fine paid after a pattern of harm is not equivalent to stopping the harm.
The EU AI Act’s risk-based structure is more ambitious, but enforcement capacity will decide whether it matters. Requirements that cannot be inspected are policy statements. Requirements that produce logs, audits, disclosures, and remedies can change corporate behavior.
For the industry
The commercial incentives currently favour spreading accountability. Vendors sell scalable systems. Buyers want efficiency. Lawyers can blame the tool. Executives can describe bad outcomes as model limitations.
The durable advantage will belong to firms that make responsibility legible. In insurance, healthcare, credit, and public services, the ability to explain, correct, and compensate will become core infrastructure—not customer service added after deployment.
What I’m watching next
First, I am watching whether the EU AI Act’s high-risk enforcement produces organisational changes rather than compliance-page changes. The useful signal will be documentation that matches actual case handling: versioned logs, meaningful overrides, vendor contracts, and records of who approved a decision.
Second, I am watching the litigation around the Air Canada chatbot. It was a small damages award, but it supplied a clear principle: a company cannot turn an official chatbot into an external contractor to evade responsibility. Similar cases involving banks, insurers, and platforms could establish that “AI agent” language does not create a legal shield.
Third, I am watching whether major model providers move from broad disclaimers toward product-level guarantees. Terms such as “use at your own risk” will not survive scrutiny in high-stakes deployments. Contracts that allocate liability for known failure modes, guarantee audit access, and preserve deployer logs will matter more than whether a model claims to be more accurate than a competing model.
I am also watching employment platforms and healthcare buyers. The key question is whether they will disclose the actual variables behind rankings and rejections, not whether they offer a friendly chatbot to explain them. A human who merely paraphrases a score has not created accountability.
The bottom line
“AI did it” is not a cause. It is a statement about the presence of automation, not about responsibility. Systems do not choose corporate objectives, approve budgets, set policy, or decide whether affected people can appeal. People and institutions do.
So stop asking only whether an AI made the decision. Ask who owned the workflow: who supplied the data, chose the objective, set the threshold, deployed the system, and could have stopped it. If that actor cannot explain the outcome, reverse it, and compensate the person harmed, then the absence of “AI did it” is not the problem. The refusal to say who did it is.
Comments
Leave a message below. Your comment saves to your browser.