That gap is the subject of this essay, and it is not a gap of conviction. Nobody in those organisations is against fairness. Nobody argued for opacity or against giving people a way to challenge a decision that ruined their week. The framework was written by capable people, reviewed by capable people, and approved by a committee that agreed with every word of it.

And then, for the most part, nothing happened.

The Finding That Explains the Gap

A study of 3,000 companies, published by UNESCO and the Thomson Reuters Foundation, sought to uncover the mechanics behind the adoption statistics. Two of its findings sit together in a way that should stop anyone who has ever signed off on a governance framework.

Only about one in six or seven companies could identify the person within their organisation responsible for the ethical risks arising across the AI lifecycle.

And for roughly three-quarters of companies, there was no evidence of a policy on the quality of the training data used by their AI.

Put those together. Most organisations cannot name who is accountable, and most have not addressed the thing the previous essay in this series called the ground on which other controls stand. These are not separate failures. The second is a consequence of the first. Data provenance did not get done because it was nobody’s — and a task that belongs to everybody, in an organisation of any size, belongs to no one at all.

A responsibility distributed across four functions is not shared. It is lost. Everyone assumes it sits slightly to their left.

The Shape of the Thing That Doesn’t Get Done

Watch how a governance framework typically arrives, because the failure is built into the delivery.

It is a single document, or a suite of them, covering the whole territory: principles, risk, data, fairness, explainability, human oversight, incident response, procurement, monitoring, board reporting. It is comprehensive. Comprehensiveness is the thing it is most proud of, and the thing that will kill it.

Because when it lands, every function reads the section that concerns it and correctly concludes that it cannot act alone. Data governance needs engineering. Engineering needs the model standard. The model standard needs the risk appetite. The risk appetite needs the board. The board needs a paper from the executive who is awaiting the operationalisation of the framework. Everybody is waiting for a prerequisite that belongs to someone else, and every one of them is being reasonable.

So the framework is adopted in the specific sense that a committee has agreed on it and it now exists. And nothing about how any decision gets made on Tuesday has changed.

This is what I mean by a monolith. Not that it is long — length is fine — but that it has no entry point. It cannot be started, only completed, and so it is never started.

Divide by Owner, Not by Topic

The instinctive fix is to break the framework into pieces, and organisations do this constantly. They almost always break it the wrong way.

The wrong way is by topic. A fairness workstream, an explainability workstream, a data workstream, a monitoring workstream. It looks like decomposition, and it changes nothing because a topic is not something anybody owns. Fairness is not a department. It has no budget, no head, no seat at the executive table, no line in anyone’s objectives. A fairness workstream is a coordination problem with a name — it needs the data people, the modelling people, the risk people, the legal people, and the business owner, none of whom report to it. You have not divided the work. You have divided the document and left the work exactly where it was.

The right way is by an accountable owner. Cut the framework along the seams that already exist in the organisation — the places where authority, budget, and a named executive already sit. Not fairness and explainability, but the thing the board must decide and cannot delegate. The thing the risk function enforces before deployment. The thing the technology function builds into the pipeline. The thing the general counsel owes to a regulator. The thing procurement must extract from a vendor before signing. The thing that the operating business runs every day.

Each of those already has a name associated with it. Each has a budget. Each can begin on Monday without waiting for the others, since the owner has the authority to start.

Divide governance by topic, and you get a coordination problem nobody owns. Divide it by owner, and you get a set of programmes that can each begin on Monday.

Why the Seams Matter More Than the Content

This is not an argument about how to organise a filing cabinet. The seams determine whether anything moves, and they do so for three reasons unrelated to the quality of the governance content.

Different owners move at different speeds. A board sets risk appetite annually. A technology function ships fortnightly. A procurement cycle turns over when a contract renews. A framework that requires all three to move in lockstep moves at the pace of the slowest, which in practice means it does not move. Cut along the ownership seam, and each part runs at its natural clock. The board can ratify a prohibition list while engineering is still building the logging standard, and neither is blocked by the other.

Different owners respond to different pressures. The general counsel is under pressure due to regulatory exposure. The technology executive is driven by delivery. The chief risk officer is pushed by the board. These pressures are real, and a framework that ignores them fails; a framework that follows the ownership seams can attach each obligation to the pressure that will actually cause it to be honoured.

And an owner can be held to it. This is the decisive one. The whole series has argued that governance without a named accountable human is theatre — the royal commission that had to excavate who was responsible; the vendor’s data scientist who chose the fairness metric; the reviewer with no authority to overturn. A framework divided by topic reproduces exactly that failure at the level of the framework itself. It creates obligations with no owner and then expresses surprise when they are not met.

Separable, Not Fragmented

There is an obvious objection here, and it is the right one: this sounds like a recipe for fragmentation. Six separate programmes, six owners, six interpretations, six sets of thresholds, and an organisation whose AI governance is a patchwork that agrees with itself nowhere.

That objection is correct about the risk, and it identifies precisely what has to be got right. Separability without a spine is fragmentation. What prevents it is that the division occurs in only one layer.

Divide the doctrine. Unify the enforcement.

The constitutional layer — what the organisation will not do, what its risk appetite is, who is ultimately accountable — is single, board-owned, and a prerequisite to everything else. It is not one of the divisible pieces. It is the root from which the divisible pieces derive their authority, and no workstream may contradict it or reinterpret it locally.

And at the other end, the enforcement point is single too. One gate that a system must pass before it goes into production. One register of what is deployed and under whose authority. One escalation path when something breaks. One board line where it is all reported. The workstreams feed that gate; they do not each build their own.

What is divisible is the middle — the standards, the processes, the tooling, the day-to-day machinery of each domain. That can proceed at six different speeds, under six different owners, with six different budgets, and still compose, because everything at the top is common and everything at the bottom converges.

Divide the doctrine, unify the gate. Anything that divides the enforcement point does not decompose the framework. It has abandoned it.

What This Buys You

Consider what becomes possible once the division follows ownership.

An organisation can start where its pain is. A general counsel exposed to regulatory explainability can build that, alone, now, without waiting for the fairness programme to be scoped. A technology executive who knows the deployment gate is the real problem can build the gate. A board that has looked at the last three essays in this series and gone slightly pale can ratify a prohibition list and a set of accountability assignments in a single meeting, and that ratification is immediately load-bearing for everything built afterwards.

And they can run in parallel, because none of them is a prerequisite for the others except the constitutional layer, which is a prerequisite for all of them and takes a board meeting rather than a programme.

This is the difference between a framework that yields value on a horizon of weeks and one that yields nothing until the day it is finished — which, since it is never finished, is never. The evidence bears this out plainly: the organisations with the highest governance maturity are consistently those with clear, assigned ownership. Not the ones with the best documents. The ones where somebody’s name is against the thing.

The Honest Cost

This approach has a price, and a framework author who claims otherwise is selling something.

Six owners mean six interpretations to reconcile, and reconciliation is work that a monolith does not require. There will be duplication at the boundaries — two workstreams both touching model documentation, two both touching vendor obligations — and someone senior has to arbitrate when they collide. The common layer at the top and the common gate at the bottom have to be maintained against six programmes that will, under delivery pressure, each be tempted to build a local variant that suits them better. That temptation has to be refused every single time, because the first exception is the end of the architecture.

That is real overhead, and it is the price of adoptability. The monolith has none of it — no boundary disputes, no reconciliation, perfect internal consistency — and it achieves this by never being implemented. An internally consistent framework that nobody has started is not a superior artefact. It is a document.

Everybody Agreed

Return to the number this essay opened with, because it is not really a statistic about frameworks. It is a statistic about how organisations fail to act on things they sincerely believe.

Eighty-seven per cent have the framework. Fewer than a quarter run it. Nobody in that gap is a villain — and by now that sentence should sound familiar, because it has been true of every case in this series. Nobody at 7-Eleven decided to conduct biometric surveillance on 1.6 million customers. Nobody at Northpointe set out to double the false-positive rate for Black defendants. Nobody in the Dutch tax authority intended to take 1,600 children from their parents. Nobody downloaded child abuse material on purpose.

In every case, the harm occurred through a gap where a named human being should have been standing, and in every case, the organisation had documents stating otherwise.

The framework is not the governance. The framework is a description of governance that someone still has to build — and they will only build it if the thing they were handed can be started by someone with the authority to do so on a Monday, without permission from four other people who are all waiting for each other.

Everybody agreed. That was never the hard part. The hard part is that agreement, distributed evenly across an organisation and attached to nobody in particular, is indistinguishable from nothing at all.

References

This piece draws on published survey research into AI governance adoption and implementation.

1. UNESCO and Thomson Reuters Foundation. Responsible AI in Practice: 2025 Global Insights from the AI Company Data Initiative. Based on public data from 3,000 companies, collected July–November 2025.

2. McKinsey & Company. State of AI Trust in 2026: Shifting to the Agentic Era. AI Trust Maturity Survey, approximately 500 organisations, December 2025 – January 2026.

3. PwC. 2025 US Responsible AI Survey. Survey of 310 US business leaders, September–October 2025.

4. AuditBoard. From Blueprint to Reality: Execute Effective AI Governance in a Volatile Landscape. Survey of over 400 GRC and audit professionals, 2025.

5. Stanford Institute for Human-Centered AI. AI Index Report 2026, Responsible AI chapter.

In December 2023, a researcher at the Stanford Internet Observatory ran perceptual and cryptographic hash matching across it. He found 3,226 suspected instances of child sexual abuse material. More than a thousand were externally validated as known CSAMs.

LAION took the dataset down. The Stanford report recommended that models trained on it be deprecated and their distribution ceased where feasible.

And here is the sentence that ought to change how every organisation thinks about the data under its AI. The lead author put it plainly: if you downloaded that dataset for any purpose, you have CSAM unless you took extraordinary measures to prevent it.

Nobody had known. Not LAION, which built it. Not the companies that trained on it. Not the thousands of businesses that shipped products built on top of models trained on it. Every one of them would have told you, in complete good faith, that their training data was fine — because not one of them could see.

The Question Nobody Can Answer

Pick any AI system in your organisation that makes decisions about people. Now ask the question that sounds trivially simple and turns out not to be.

Where did its training data come from?

Not roughly. Specifically. Which datasets, obtained from whom, collected under what basis, transformed how, by whom, and when. If some of it is personal information, on what lawful footing was it collected — and does that footing extend to training a model, or merely to the purpose for which it was originally gathered? If any of it was bought, what did the vendor actually warrant, and can they evidence it? If the system is built on a foundation model, what was that model trained on?

In most organisations, this question produces silence, followed by a series of partial answers that trail off. The data science team knows what it pulled. The data engineering team knows what it ingested. Nobody has the whole chain, because nobody was ever asked to keep one — and the model works, so the question never became urgent.

Then the system goes into production and starts deciding things about people, and every claim the organisation makes about it quietly rests on a fact nobody has established.

What You Are Actually Claiming

Consider what an organisation asserts when it says its AI is well governed. Then notice what each of those assertions silently assumes about the data.

It says the system is fair — that it has measured disparities across groups and they fall inside an acceptable threshold. But a fairness measurement is a statement about a population. It says: within the people this system affects, outcomes do not differ unacceptably by protected characteristic. If the organisation cannot characterise the population its training data actually represents — who is in it, who is missing, what proportion of each group — then the fairness metric is a number computed over an unknown. It is precise, reproducible, green on the dashboard, and unanchored to anything.

It says the system is explainable — that it can name the principal factors that drove any given decision. But an explanation is a claim about what the model learned. If nobody can say what it learned from, the explanation is a description of the model’s behaviour dressed as an account of its reasoning. It may be perfectly accurate about what the model did. It cannot be vouched for as an account of why.

It says a person can contest the decision — that a reviewer can look again at the specific basis on which it was made. But a review reconstructs a decision from its inputs. If the provenance of those inputs is unknown, the reviewer is examining a record whose reliability the organisation cannot attest to itself.

Unknown data does not simply fail the data test. It silently voids the fairness test, the explanation, and the review — each of which will still run, still pass, and still report green.

The Failure That Leaves No Trace

This is what makes the provenance gap more dangerous than the failures in the earlier essays in this series, and it is worth being precise about why.

A prohibited use announces itself eventually — someone finds out you were scanning faces. A fairness breach shows up as a measurable disparity. An absent explanation produces a complaint. A rubber-stamp review can be exposed by calculating an override rate.

A provenance gap does none of this. It produces no symptoms. The model trains. The metrics compute. The dashboards go green. The audit passes because the audit checks whether the controls were run, not whether the ground beneath them was ever verified. An organisation can operate for years in full compliance with its own framework on data whose origin nobody can state — and nothing will ever surface it.

Until something does. And when it does, it does not arrive as a data-quality issue. It arrives as a regulator asking whether you had consent, or a court asking what the model learned, or a research team publishing a paper about what was in the dataset all along.

LAION did not degrade. It was always what it was. What changed in December 2023 was that somebody finally looked.

The Part You Cannot Fix With Diligence

There is a version of this problem that careful work solves. Document your sources. Keep a lineage. Check your consent basis. Assess your data quality. All of this is achievable, and an organisation that does it is in a far better position than one that does not.

And then there is the version that careful work does not solve, and it is now the version most organisations are in.

If your system is built on a foundation model — and increasingly, everything is — then a material part of its behaviour was determined by a training corpus that the provider will not fully disclose and that you cannot inspect. This is not a failure of your diligence. It is not something a better vendor questionnaire will fix. The information is structurally unavailable to you, and no amount of contractual language creates knowledge that nobody has.

So the honest position for most enterprises today is this: there is a layer beneath your AI whose provenance you cannot establish, and you are accountable for its behaviour anyway.

The vendor’s opacity does not transfer the obligation. It simply means you have built on ground you are not permitted to inspect, and you remain answerable for what is buried in it.

This does not eliminate the obligation, nor is it an excuse. It means the unknown has to be named, assessed for its specific use, and signed off by someone with the authority to accept it — rather than silently absorbed into a system that is then described as governed.

What Knowing Your Data Actually Means

The requirement is not a folder of documentation. It is the ability to answer four questions about every material dataset under a consequential system, on demand, without a project.

1. Where it came from. The origin, the collection method, and the chain of custody and transformation from source to the form the model was trained on. Not a vague sense of the source. A lineage you could put in front of a regulator.

2. On what basis? If it contains personal information, there must be a specific lawful basis for using it to train this model. Consent to collect data for one purpose is not consent to train a model on it for another, and the fact that everybody does it does not make it so.

3. Whether it fits. Whether the data actually represents the population the system will decide about — completeness, currency, and representativeness — depends on the model, because a model trained on a population that excludes the people it will judge is not a slightly worse model. It is a model whose errors are systematic, invisible, and concentrated on the people least able to complain.

4. Whether it still holds. Provenance is not a certificate you obtain once. A consent basis can lapse. A data-sharing right can terminate. A dataset can go stale as the world it describes moves on. The question is not whether you knew your data at the time of deployment. It is whether you know it now.

The Uncomfortable Audit

Here is an exercise that costs nothing and tends to be clarifying, in the way a diagnosis is.

Take your three most consequential AI systems. For each, ask the team to produce, within a week, the documented provenance of every material dataset it was trained on — origin, custody, consent basis, and quality assessment.

Watch what comes back.

In most organisations, some of it will arrive promptly and in good order. Some will arrive after a scramble, reconstructed from memory and commit logs. And some will not arrive at all, because the person who built that pipeline has left, or the vendor cannot say, or the dataset was assembled in 2019 by someone who did not think anyone would ever ask.

That last category is the finding. Not the volume of it — the existence of it. Because those are the systems that are currently deciding things about people, on a basis the organisation cannot state, while every control built on top of them reports that everything is fine.

The Ground Under Everything

This series has worked through the things a responsible organisation owes the people its systems judge. A prohibition on the uses it will not make. A fairness threshold someone chose and owns. An explanation the affected person can actually use. A right to challenge that a human with real authority can honour.

Every one of them assumes you know what the system was built on.

Data provenance is not the fourth or fifth item on a responsible-AI checklist, to be addressed only after the visible controls are in place. It is the ground the other controls stand on, and it is the only one whose absence produces no symptoms at all — no complaint, no discrepancy, no failed audit. Just a set of green ticks over a foundation nobody checked.

Somebody, somewhere, assembled the data your model learned from. They made choices about what to include and what to leave out, and those choices are now inside the decisions your organisation makes about people’s money, their housing, their employment, their liberty.

If you cannot say who they were or what they chose, then you are not governing that system. You are operating it and hoping.

References

This piece draws on the Stanford Internet Observatory investigation into LAION-5B and reporting on its consequences.

1. Thiel, D. Identifying and Eliminating CSAM in Generative ML Training Data and Models. Stanford Internet Observatory, Stanford University, December 2023.

2. Schuhmann, C., et al. LAION-5B: An Open Large-Scale Dataset for Training Next Generation Image-Text Models. Advances in Neural Information Processing Systems (NeurIPS), 2022.

3. Reporting on the removal of LAION-5B, 404 Media and TechCrunch, December 2023 and August 2024.

4. Office of the Australian Information Commissioner. Determination on Clearview AI, Inc. Canberra, 14 October 2021.

When the model flagged a family, their benefits were suspended, and the tax authority demanded repayment of everything they had received, sometimes going back years. The sums ranged from 20,000 euros to 60,000 euros, payable in full, with no instalment plan. The trigger was often a missing signature or an unticked box.

Around 26,000 families were wrongly accused. Some estimates put it above 35,000. Parents were bankrupted, lost their homes, and lost their jobs. More than sixteen hundred children were taken into foster care. Some parents took their own lives.

The parliamentary inquiry that finally examined all this gave its report a title that does not require translation: Ongekend Onrecht. Unprecedented Injustice. It found that the tax authority had violated fundamental principles of the rule of law, and that fraud investigations had been triggered by something as simple as an administrative error, with no malicious intent on anyone’s part.

On 15 January 2021, the entire Dutch cabinet resigned.

The Sentence That Explains Everything

Buried in the analysis of how this happened is a sentence that ought to be printed and pinned above every desk where an AI system is being designed.

The risk score was not a tool to support a caseworker’s judgment. It was treated as a judgment.

The Dutch tax authority was not short of humans. There were caseworkers. There were supervisors. There were managers, directors, a state secretary, a minister, and a cabinet. The system did not operate in a vacuum, and at no point did anyone claim that a machine had autonomously ruined 30,000 families while the civil service watched, powerless.

Humans were in the loop at every stage. Humans read the flags. Humans sent the letters. Humans received the phone calls from parents who could not understand what they had done wrong, and humans told them to pay.

Somebody looked at it. Nobody changed it.

The Control That Is Not a Control

Every organisation deploying consequential AI offers the same reassurance, and it is offered so reflexively that it has stopped being examined: there is a human in the loop.

It is the most comforting sentence in AI governance and the least informative. It describes a position on an org chart. It says nothing whatsoever about whether that person can, in practice, do anything.

Consider what being in the loop actually meant for a Dutch caseworker. They saw what the system surfaced — a risk score, a flag, a case marked for recovery. They did not see the model, could not interrogate its reasoning, and had no way to know that dual nationality was among the variables driving the score in front of them. They had a queue. They had targets. The institutional culture around them, as the inquiry found, was fraud-first: the presumption ran against the citizen.

Now ask what it would have taken for that person to overturn the machine. They would have had to form an independent view against a system the organisation trusted, justify it to a supervisor, absorb the delay in their own throughput, and accept the professional risk of being the person who let a suspected fraudster through. Agreeing with the system cost nothing and took seconds. Disagreeing costs time, standing, and nerve.

Put a competent, decent person in that position, and they will agree with the machine. Not because they are lazy or cruel, but because every force acting on them points that way. The review is real. The authority is not.

Human presence is not human authority. A person who reviews what the machine surfaced under a clock, with no power to see inside it and no protection from being overruled, is not oversight. They are a signature.

Why This Control Is the Easiest to Fake

Every other control in AI governance leaves evidence. A fairness threshold is a number. An audit trail has records. A prohibited-use screen has a decision logged under a name. If those do not exist, their absence is visible.

Human review leaves evidence of exactly the same kind, whether it is real or theatrical. A case was reviewed. A person’s name is against it. A timestamp exists. The box is ticked. And a review in which the human agreed with the machine because agreeing was the path of least resistance looks, in every record the organisation keeps, identical to a review in which the human independently reached the same conclusion.

This is why it is the easiest control to fake — and, more troubling, the easiest to fake unintentionally. Nobody has to decide to build a rubber stamp. You get one for free, by default, simply by placing a human in a process without giving them the time, the information, the standing, or the protection they would need to disagree.

The organisation then reports, in complete good faith, that its automated decisions are subject to human review. The auditor confirms it. The regulator accepts it. And the override rate, if anyone ever calculated it, would be indistinguishable from zero.

An organisation that cannot tell you how often its human reviewers overturn the machine does not have human oversight. It has a person and a hope.

The Number Nobody Calculates

So here is a question worth asking inside your own organisation, and it takes about a day to answer.

For every AI system that makes or shapes decisions about people and that has a human review step, how often does that human change the outcome?

Not how many cases were reviewed. How many were reversed?

If the answer is that nobody has ever calculated it, that is itself the finding, and it should be treated as one. It means the control has been claimed, relied upon, reported to a board, and never once tested. It means the organisation has been assuring itself for years that a mechanism’s effectiveness has never been measured.

And if the number comes back at 1-2%, resist the comfortable interpretation. The comfortable interpretation is that the model is simply very good. The uncomfortable one is that the reviewers cannot see enough to disagree, do not have time to disagree, or have learned that disagreeing is not rewarded. Both interpretations produce the same statistic. Only one of them is oversight.

The distinguishing test is not the rate. It is what happens when a reviewer overturns the machine. Is that treated as the control working — as evidence that the system caught an error before it reached a person? Or does it generate a query, a variance report, a quiet conversation about why this officer’s numbers look different from everyone else’s?

Organisations tend to know the answer to that question immediately, and to be uncomfortable saying it out loud.

What a Real Review Needs

A contestation review that can actually overturn a machine requires four things, and it fails if it lacks any one of them.

• Independence. The review cannot be conducted by the process that produced the decision, nor can it consist of re-running the model. A second look at the same output by the same system is not a second look; it is the first decision defending itself.

• The actual record. The reviewer must see the specific facts of this decision — the factors that drove it, the inputs it used, the thresholds it applied — not a general account of how the system behaves. A reviewer working from the machine’s own summary is reasoning inside the machine’s frame and will almost always confirm it.

• Real authority, exercised. The power to uphold, modify, or overturn — with the time to use it, the standing to survive using it, and no penalty attached. This is the one that quietly fails, not by being removed but by being made expensive.

• A route in. None of the above matters if the affected person cannot access the process at all. The Dutch families were, according to the inquiry’s findings, frequently given no explanation, no appeal pathway, and no evidence beyond the score — and were then required to prove their innocence of a fraud they had not been told they were suspected of, using documents the tax service had often lost.

That is the inverted burden of proof, and it is the signature of a system that has stopped treating its output as a proposal and started treating it as a finding of fact.

The Objection

A fair objection: real review at scale is impossible. A bank makes millions of automated decisions. A tax authority processes tens of thousands of benefit claims. You cannot have a human independently re-adjudicate each one — that would defeat the purpose of automating anything and would cost more than the system’s entire benefit.

This is correct, and it is not the argument.

Nobody is proposing that every automated decision receive a substantive human review. The proposal is narrower and entirely affordable: that any decision a person challenges receives one. The machine decides by default. A human decides on a challenge. That is the whole design, and the volume it generates is a small fraction of the volume the system handles, because most people do not contest most decisions.

What makes the volume manageable is precisely what makes the right meaningful — it is exercised by the people who believe the machine got it wrong about them. Those are the cases where a second look has the highest value, and where the second look is currently most likely to be a formality.

The Dutch tax authority did not fail because it could not review 26,000 cases. It failed because when parents rang up to say the machine was wrong about them, nobody who answered the phone had the authority to agree.

The Last Thing Standing

Contestation is the final control and the one that matters most, because it is the only one that operates after everything else has failed.

The prohibited-use screen can miss a use. The fairness threshold can be set incorrectly. The explanation can be inadequate. The data can be flawed in ways nobody detected. Every upstream control is fallible, and in any system of sufficient scale, some of them will fail some of the time.

The right to contest is what stands between that failure and the person it is about to harm. It is the moment at which a human being can look at what the machine has decided and say: not this one, not this person, not today.

Which is why an organisation that has built every other control and left this one hollow has built nothing at all. The machine will be wrong. It is wrong for somebody right now. The only question that matters is whether, when that person picks up the phone, anyone on the other end can actually help them.

In the Netherlands, for eight years, there was not. Somebody looked at each of those cases. Nobody changed them. And by the time anybody with authority did look — properly, with the power to act — 26,000 families had already been destroyed, sixteen hundred children were in foster care, and a government fell.

References

This piece draws on the Dutch parliamentary inquiry, findings of the Dutch Data Protection Authority, and reporting on the toeslagenaffaire.

1. Tweede Kamer der Staten-Generaal. Ongekend Onrecht: Verslag Parlementaire Ondervragingscommissie Kinderopvangtoeslag. The Hague, 17 December 2020.

2. Autoriteit Persoonsgegevens. Findings on the processing of nationality data by the Belastingdienst. The Hague, 17 July 2020.

3. Nationale Ombudsman. Geen powerplay maar fair play. The Hague, 2017.

4. Reporting on the toeslagenaffaire by Trouw and RTL Nieuws, 2018–2021.

5. Henley, J. Dutch government faces collapse over child benefits scandal. The Guardian, 14 January 2021.

He called Goldman Sachs, which underwrote the card, and asked why. The representatives could not tell him. They told him, repeatedly, that it was the algorithm. He posted about it. Steve Wozniak replied that the same thing had happened to him and his wife. The New York Department of Financial Services opened an investigation.

Then Goldman raised his wife’s credit limit. It did so without requesting a single additional document and without addressing why the limit had been low in the first place.

The regulator reviewed underwriting data for approximately 400,000 New York applicants. It found no unlawful discrimination. Applications from women and men with similar credit characteristics generally received similar outcomes. And it described the bank’s credit decisions in three words that are worth reading slowly.

Explainable. Lawful. Consistent with the bank’s credit policy.

Explainable to Whom?

Hold both of those facts in your head at once, because they are both true and they are the whole argument.

The system was explainable. A regulator with subpoena powers, full access to the model, the underwriting data, thousands of pages of records, and months of investigation could reconstruct exactly why any given credit limit was what it was. The bank could show its work. The decisions held up.

And the woman who called to ask why she had been given a twentieth of her husband’s limit, on a better credit score, was told that it was the algorithm.

Both of those describe the same system. The system was, in the regulator’s words, explainable — and the person it had judged could not obtain an explanation. Nothing here is a contradiction. It is simply that the word explainable was doing entirely different work in the two sentences, and almost nobody noticed.

A system can be fully explainable to a regulator and completely opaque to the person it just refused. Those are not degrees of the same thing. They are different obligations to different people, and meeting one says nothing about the other.

Three People Are Asking, and They Want Different Things

When a consequential AI decision is made about a person, three parties have a legitimate claim to an explanation, and what each of them needs bears almost no resemblance to what the others need.

The regulator needs a complete, tamper-evident record from which the decision can be reconstructed and defended, months or years later, under adversarial examination. It wants the model version, the inputs, the policy applied, the thresholds, and the audit trail. It is exhaustive, technical, and no ordinary person will ever read it — which is fine, because no ordinary person is its audience.

The operator — the officer, the underwriter, the case manager who sits between the machine and the customer — needs something different. They need a concise, real-time account of what the system found and how confident it is, sufficient to agree with it, question it, or override it. They have context, a queue, and about ninety seconds. An audit log is useless to them. So it is a policy document.

And the person who was refused needs something different again: what actually drove this decision about me, in language I can understand, and what would I have to change for the answer to be different? They have no technical knowledge, no access to the model, and the highest stakes of anyone in the chain. It is their credit, their housing, their job, their liberty.

Three audiences. Three explanations. Three separate obligations. And the artefact that discharges one discharges none of the others.

The One That Always Gets Built

Of the three, one is reliably built, and it is not an accident which one.

The audit trail gets built because the organisation needs it. Logging is a familiar engineering discipline. Regulators ask for it directly, and they ask with the power to fine. Internal audit wants it. Legal wants it. Its audience has standing, resources, and teeth.

So the audit trail exists, and it tends to be genuinely good. And then something quietly follows: the organisation, having built the artefact that protects itself, says that its AI is explainable — and it is telling the truth.

The operator’s explanation, meanwhile, is usually approximated. A confidence score. A list of contributing features. Something that looks like a reason without being usable as one.

And the individual’s explanation is the one that goes missing. Not because anyone decided to withhold it. Because its audience — the applicant, the customer, the citizen — has the least power to demand it, the least standing to know what they are owed, and no seat at the table where the system was scoped. The obligation is real, and the pressure to meet it is nil.

The tier that protects the organisation is the tier that gets funded. The tier that protects the person is the tier that gets deferred. This is not a coincidence; it is the shape of every incentive in the building.

It’s Just the Algorithm

Which brings us back to the phone call, because that sentence deserves proper examination.

When a Goldman representative told a customer that it was the algorithm, they were not being evasive. They were being accurate. They genuinely could not explain the decision, because nothing in the system had been built to tell them — and nothing had been built to tell the customer, because the customer’s explanation was not an artefact anyone had been required to produce.

The information existed. Somewhere in the underwriting data that the regulator would later review across 400,000 applicants, the factors that drove that specific credit limit were sitting there, recoverable. The bank was not hiding them. It simply had no mechanism that turned them into a sentence a human being could hear.

So the customer got the truest thing the representative could say: nothing at all.

And notice how the story ends. Goldman raised the wife’s credit limit without additional documentation. That is what an organisation does when it cannot explain a decision and needs the complaint to go away: it changes the outcome, rather than the process, for the person who complained loudly enough to be heard. Everybody else keeps the limit they were given and never finds out why.

What the Person Is Actually Owed

An explanation owed to an individual is not a shorter version of the audit trail. It has specific content, and its failure lies in vagueness, even when attempted.

• The principal factors. Not the model, not a feature list — the handful of things that actually drove this decision, in order, stated so a person recognises them. Your existing debt relative to your income and the length of your credit history were the main factors. Not: a range of factors was considered. That sentence explains nothing, and everyone who writes it knows so.

• The counterfactual. For an adverse decision, the smallest change that would have produced a different outcome. Had your reported income been higher by roughly this much, or had this account not been in arrears, the decision would have changed. This is what makes the explanation usable rather than merely informative. It is the difference between being told why the door is shut and being told where the handle is.

• Delivery on time matters. In language, a person without a statistics degree can act on. An explanation that comes after escalation, on request, three weeks later, has not been delivered. It has been extracted.

Why You Cannot Bolt This On

Here is the part that makes this a design problem rather than a communications problem, and it is the reason so many organisations discover it too late.

The principal factors and the counterfactual cannot be reverse-engineered from a model that was not built to produce them. The attribution has to be computed at the moment of decision and captured. The counterfactual requires the system to answer a question about a decision it did not make. If the model was chosen or bought without that capability, no amount of goodwill afterwards will conjure it — the information was either recorded when the decision was made, or it is gone.

This means explainability to the individual is not a reporting feature to be added in a later release. It is a constraint on which models you are permitted to use at all, decided before anything is built or procured. A vendor’s system that cannot explain why it made a decision is not a system with a documentation gap. It is a system that cannot be deployed against people, and the right time to discover that is in procurement, not in a phone call with a customer you cannot answer.

A black box you bought is still a black box you own. The vendor’s opacity does not transfer the obligation; it just means you have acquired a system you cannot use responsibly.

The Question to Ask Your Own Organisation

Somewhere in your organisation, there is a claim that the AI is explainable. It is probably true. The question is what it is true of.

Ask to see the explanation a person actually received after an adverse automated decision. Not the audit log. Not the model card. Not the policy. The letter, the screen, the sentence that went to the human being on the other end.

Then ask whether it names the factors that drove their outcome, and whether it tells them what would have changed it. If it says that a range of factors was considered, or that the decision was based on our credit criteria, or nothing at all, then the organisation has built the tier that protects itself and skipped the one that protects them.

And it will have done so while truthfully saying, in a regulatory filing and under oath, that its systems are explainable.

Both Things Were True

The regulator was right. The decisions were explainable, lawful, and consistent. 400,000 applications were reviewed, and no discrimination was found; the model is doing what the credit policy says it should.

The customer was also right. She could not find out why. Nobody could tell her. It was the algorithm.

There is no contradiction between those two sentences, and that is precisely the problem. An organisation can satisfy every explainability obligation it has been asked about, pass every audit, survive every investigation, and still leave the people it judges with nothing they can understand or use — and it will not even notice, because the word explainable will keep coming back green.

Explainable is not a property. It is a relationship, and it has to be true of somebody. Ask who.

References

This piece draws on the New York State Department of Financial Services’ investigation report and contemporaneous reporting on the Apple Card.

1. New York State Department of Financial Services. Report on Apple Card Investigation. Albany, March 2021.

2. New York State Department of Financial Services. DFS Issues Findings on the Apple Card and Its Underwriter Goldman Sachs Bank. Press release, 23 March 2021.

3. Reporting on the Apple Card credit limit allegations, CNBC and CNN Business, November 2019.

4. Banking Dive. Goldman cleared of bias claims in NYDFS’s Apple Card probe. 24 March 2021.

COMPAS is a risk-assessment tool sold by a private company. It takes a defendant’s history and answers to a questionnaire, then returns a score from 1 to 10 predicting the likelihood of reoffending. Judges across the United States have used those scores in decisions about bail, sentencing, and parole.

Three years later, ProPublica examined more than 10,000 COMPAS assessments in Broward County, Florida, and published a finding that became one of the most-cited results in the history of algorithmic accountability. Among defendants who did not go on to reoffend, Black defendants were nearly twice as likely as white defendants to have been labelled high risk. Among those who did reoffend, white defendants were far more likely to have been labelled low risk. The headline read: Machine Bias.

Northpointe, the company that built COMPAS, responded with a rebuttal. It said the tool was fair, and it produced numbers to prove it. Among defendants who scored a seven, roughly 60% of white defendants and 61% of Black defendants went on to reoffend. A seven meant the same thing regardless of race. The score was doing exactly what it claimed.

Here is the part that should unsettle you. Both of them were telling the truth.

Two True Answers to the Same Question

ProPublica and Northpointe were not arguing about the data. They agreed on the data. They were arguing about what the word fair means, and neither of them said so out loud, because neither of them fully realised it.

ProPublica used a definition that most people, asked on the spot, would endorse: when the system is wrong, it should be wrong equally often for everyone. If you did not reoffend, your chance of being wrongly branded high-risk should not depend on your race. By that standard, COMPAS failed, and failed badly.

Northpointe used a different definition, which most people would also endorse if you asked them separately: a score should mean the same thing for everyone. A seven should carry the same probability of reoffending whether the defendant is Black or white, or the score is lying to the judge. By that standard, COMPAS passed.

Both definitions are reasonable. Both are defensible in a courtroom. And they cannot both be satisfied.

The Mathematics That Ends the Argument

This is not a matter of opinion, or of engineering effort, or of a better model trained on cleaner data. It is a theorem.

In 2016, Kleinberg, Mullainathan and Raghavan formalised three fairness conditions at the heart of these debates and proved that, except in highly constrained special cases, no method can satisfy them simultaneously. Chouldechova proved a parallel result independently. The constrained special cases are these: either the system predicts perfectly, or the two groups have identical base rates — the same proportion of each group genuinely belongs to the class being predicted.

Perfect prediction does not exist. Equal base rates, in the domains where these systems are deployed — criminal justice, credit, employment, welfare — do not exist either, because the world the data records is not equal. Black defendants in Broward County were rearrested at a higher rate than white defendants, for reasons that have everything to do with policing and history and nothing to do with the algorithm.

And that is enough. Once the base rates differ, the mathematics closes the door. A system calibrated so that a score means the same thing across groups will produce unequal error rates. A system equalised for error rates will produce scores that mean different things across groups. You may have either. You may not have both. There is no third option, and no amount of retraining will find one.

Fairness is not one thing a system has or lacks. It is several incompatible things, and choosing among them is not a technical decision. It is a decision about who bears the cost of being wrong.

The Choice Nobody Admits Making

So somebody chose. Northpointe chose calibration. It chose to guarantee that a score means the same thing for everyone, and the price of that guarantee, paid entirely by Black defendants who would never have reoffended, was a doubled rate of being wrongly labelled dangerous to the community.

That was a moral decision of the first order. It determined who would be harmed by the system’s inevitable errors. And it was taken inside a private company, embedded in a model, and delivered to judges as a number between one and ten.

No judge chose it. No legislature chose it. No defendant was told a choice had been made at all. Eric Loomis was sentenced with reference to a score whose entire meaning depended on a fairness definition selected by a vendor, and the selection was never disclosed, debated, or defended because it was never presented as a selection. It was presented as a measurement.

This is the pattern, and it is everywhere. The choice does get made. It gets made by whichever metric the library computed by default, or by whichever definition the model happened to satisfy, or by the data scientist who picked one at 6 pm because the report was due. It gets made, and then it disappears into the number.

The question is never whether a definition of fairness was chosen. One always was. The question is whether anyone in your organisation knows which, and whether the person who chose it had the standing to make that choice on your behalf.

What This Means for Your Dashboard

Somewhere in most large organisations deploying AI, there is a slide with a fairness metric on it, and the metric is green.

Ask three questions about that slide.

1. Which definition of fairness is that number measuring? If nobody in the room can answer, the number means nothing — not because it is wrong, but because it is unanchored. It is the answer to a question no one has stated.

2. What does it look like in the other definitions? A model that passes on calibration but fails on error-rate balance is not a fair model, even by technical standards. It is a model that has made a specific choice about who bears responsibility for its mistakes, and the metric of failure is the receipt.

3. Who chose the threshold? Not the metric — the threshold. Someone decided that a disparity of five per cent was acceptable and eight per cent was not, and that decision determined how many people would be harmed before anyone was required to act. That number did not come from the data. It came from a person, or, more often, from nobody at all — inherited from a template, a vendor default, or a figure that seemed reasonable in a meeting.

If the answer to all three is a shrug, the organisation has not measured its fairness. It has adopted somebody else’s answer to a question it never realised it was asking.

The Objection Worth Taking Seriously

There is a serious reply to all of this, and it goes something like: if fairness is genuinely impossible to achieve on all definitions at once, then any choice is arbitrary, and this whole discourse is a counsel of despair. The engineers are stuck with an unsolvable problem, and the critics have nothing to offer but the observation that it is unsolvable.

That reply mistakes the argument. The impossibility is not an excuse; it is a reallocation of responsibility. The mathematics tells you that a trade-off must be made. It does not tell you who should make it, and that is precisely the point — because the answer, currently, is a vendor’s data scientist or nobody.

A trade-off that must be made can still be made well or badly. Made well, it is made in the open, by someone with the authority and the standing to decide who bears the cost of error in this particular domain, with the reasons written down and the losing metric reported alongside the winning one. Made badly, it is made silently, by default, and reported to the board as a fact about the world.

The impossibility theorem does not absolve anyone. It identifies the decision that has to be owned.

What Owning It Looks Like

For any consequential system, three things should be written down before it is deployed and available afterwards to anyone who asks.

• The primary fairness definition, named in advance. Not chosen after the results are in — because a metric chosen after the results are in is a metric chosen to pass. Named beforehand, on the basis of the decision the system makes and the errors that matter most in that domain. In criminal justice, a false positive costs a person their liberty; in medical screening, a false negative costs a person their life. These are not the same, and the definition should follow the harm.

• The threshold, and the name of whoever set it. The maximum disparity the organisation will accept is tighter where the stakes are higher, set deliberately rather than inherited. And attached to a human being who can be asked why that number and not a different one.

• The trade-off, stated. Where the system satisfies its primary definition and breaches another — which it will, because the mathematics guarantees it — the breach is documented, not buried. Who is bearing the cost of this system’s errors, and by how much, and why did we decide that was the right allocation? That is the sentence most fairness reports do not contain, and it is the only one that matters.

A fairness metric with no named definition, no owner, and no disclosed trade-off is not a measurement. It is a green tick over a decision nobody admits making.

Back to the Courtroom

Eric Loomis appealed, arguing, among other things, that he could not challenge a score whose workings were a trade secret. The Wisconsin Supreme Court upheld the sentence, with cautions about how such scores should be used. The tool remained in service.

Nothing in that story turns on anybody behaving badly. Northpointe built a calibrated instrument and said so. The judge used a number he had been given. ProPublica did the analysis nobody else had done. The mathematics did what mathematics does.

What failed was the assumption underlying all of it — that fairness is a property a system can be tested for and certified to have, rather than a choice about who gets hurt when the system is wrong. That assumption allows a vendor to make an enormous value judgment, encode it in a product, and pass it to a judge as a fact.

Your model has made the same choice. It made it the moment it was trained, whether or not anybody in your organisation was paying attention. The only open question is whether you can say which choice it was, who made it, and why.

Somebody chose this number. If it wasn’t you, find out who.

References

This piece draws on the ProPublica investigation, the Northpointe response, and the formal literature on fairness impossibility.

1. Angwin, J., Larson, J., Mattu, S., and Kirchner, L. Machine Bias. ProPublica, 23 May 2016.

2. Larson, J., Mattu, S., Kirchner, L., and Angwin, J. How We Analyzed the COMPAS Recidivism Algorithm. ProPublica, 23 May 2016.

3. Dieterich, W., Mendoza, C., and Brennan, T. COMPAS Risk Scales: Demonstrating Accuracy Equity and Predictive Parity. Northpointe Inc., July 2016.

4. Kleinberg, J., Mullainathan, S., and Raghavan, M. Inherent Trade-Offs in the Fair Determination of Risk Scores. Proceedings of the 8th Innovations in Theoretical Computer Science Conference (ITCS), 2017.

5. Chouldechova, A. Fair Prediction with Disparate Impact: A Study of Bias in Recidivism Prediction Instruments. Big Data, vol. 5, no. 2, 2017.

6. State v. Loomis, 881 N.W.2d 749 (Wis. 2016).

Each image was uploaded to a server and converted into a faceprint — an algorithmic representation of that person’s face, unique to them, used for biometric identification. The faceprints were compared against one another. They were also analysed to infer the customer’s age and gender.

Over the first ten months, 1.6 million surveys were completed.

The purpose of all this was to prevent staff from gaming their own customer satisfaction scores by completing the survey more than once.

The Detail That Should Stop You

When the Privacy Commissioner examined this, she made two findings that are worth holding still and looking at.

The first was that the benefit to the business was not proportionate to the impact on people’s privacy. Biometric information, as she put it, is unique to an individual and cannot normally be changed. You can reissue a password. You cannot reissue a face.

The second was quieter and, I think, more damning. If 7-Eleven wanted to know the age and gender of its survey respondents, it could have asked them. There was a survey. The questions were right there.

There is a third detail that did not make the headlines. When the regulator asked 7-Eleven how many fraudulent survey responses the facial recognition system had actually caught, the company could not say.

Biometric surveillance of 1.6 million people was deployed to protect the integrity of a customer feedback form, with no evidence that it was even working. This is not villainy. It is the absence of a question.

Nobody in That Building Was a Villain

It is important to be precise about what went wrong here, because the comfortable reading — that a company decided to spy on its customers — is almost certainly false, and the true version is far more alarming.

Picture how this actually happened. A vendor offered a customer-feedback platform. The platform had a feature that could detect duplicate responses using the tablet’s built-in camera. The feature solved a real and irritating problem: store staff were inflating their own scores. Somebody in operations thought that was clever. Somebody in procurement bought it. Somebody in IT deployed it to 700 stores. Signs went up at the door. Everyone involved was doing their job competently.

At no point in that chain was there a step that required anyone to ask: should we be taking biometric readings of every customer who touches this screen?

Not can we — that was answered by the vendor’s feature list. Not only is it legal — nobody seems to have asked that either — but the signage suggests someone thought they had covered it. The question that was never asked was whether this is a thing the organisation is prepared to do to the people who walk into its shops.

There was no villain. There was a gap in the process where a question should have been.

The Question That Never Gets Asked

Every organisation deploying AI has a mechanism for asking whether a system works. Most have one for asking whether it is safe, accurate, compliant, or profitable. Almost none have a mechanism for asking whether the use is one they are willing to make at all.

This is not an oversight. It is a structural consequence of how these decisions arrive. A capability shows up — in a vendor’s product, in a model’s release notes, in an engineer’s proof of concept — and it arrives already answering the question of feasibility. The demonstration is the answer. What follows is a conversation about implementation, risk, cost, and timeline, all of which take for granted that the thing will be done.

The prior question, the one about permission rather than possibility, has no natural moment in that sequence. Nobody’s job is to ask it. No gate requires it. No document captures it. And so it is asked, if at all, by whoever in the room has the standing and the nerve to say something that sounds naive: should we be doing this?

Feasibility arrives with its own answer. Permission has to be asked for, and if no step in the process demands it, no one will.

Which is why it goes unasked in rooms full of thoughtful, decent people. The silence is not a moral failure. It is process design.

Why Risk Management Cannot Catch It

The standard reply is that this is what risk assessment is for. Run the facial recognition system through a privacy impact assessment, and the problem surfaces.

Sometimes it does. But there is a category error hiding in that reply, and it is the heart of this essay.

Risk management asks how likely a harm is, how severe it would be, and what controls would bring it to an acceptable level. It is a machine for converting a hazard into a manageable quantity. Feed it a facial recognition system, and it will do exactly that: it will identify privacy risk, propose signage, propose encryption, propose a retention limit of seven days, propose a third-party processing agreement — and it will produce, at the end, a residual risk rating that somebody with sufficient seniority can accept.

Every one of those controls is sensible. 7-Eleven had most of them. The images were encrypted. They were held briefly. There was signage at the door. A privacy assessment, competently done, would have produced precisely this list, and the system would have gone live with a moderate residual risk, accepted by someone with the authority to accept it.

Because that is what risk machinery does. It does not have an output that says, “This is not a thing we do”. Its outputs are accept, mitigate, transfer, and avoid — and, in practice, avoid is the option nobody selects once a project has a budget and a launch date.

Put a prohibition through a risk process, and it comes out the other side as a risk. And a risk, however severe, is something a sufficiently senior person can accept.

What a Prohibition Actually Is

A prohibition is not a very high risk. It is a different kind of object entirely, and confusing the two is how organisations end up doing things they would never have defended if anyone had said them out loud.

A risk is something you weigh. You put the benefit on one side and the harm on the other; you consider the controls, and you make a judgment. This is the correct way to handle most things. It is how we decide whether to launch a product, extend a credit line, or run a model in production.

A prohibition is something you do not weigh, because weighing it is already a mistake. It is a use the organisation has decided it will not make, and the decision was taken in advance precisely so that it could not be relitigated in a room where someone has a deadline and a business case. The whole function of deciding beforehand is to remove the decision from the pressure that would distort it.

Consider what it would mean for a bank to treat this properly. Not: we will assess the risks of using AI to infer customers’ health conditions from their transaction data and apply appropriate controls. But we do not do that. There is no control environment that makes it acceptable, no benefit that offsets it, no seniority sufficient to approve it. The question does not reach the risk committee because the answer was settled before the question could be asked.

That is a prohibition. And an organisation that has none has not made a considered decision to permit everything. It has simply never noticed that some things are not risks.

The Screen Before the Gate

In practice, this means a prohibited-use screen must sit before the risk assessment, not inside it — and it has to be binary.

Before anyone assesses how a proposed use might be controlled, someone has to answer a prior question with a yes or a no: is this a use we are prepared to make? If the answer is no, the assessment stops. There is no residual risk to accept because there is no risk conversation to have. The system is not built. Nothing about the strength of the business case reopens it.

This is unfashionable because it sounds inflexible, and inflexibility is not how modern organisations like to describe themselves. But inflexibility is the entire point. A prohibition that bends under sufficient commercial pressure is not a prohibition; it is a preference that has not yet met its price. The value of the line is precisely that it does not move when moving it would be convenient — which is the only moment at which anyone ever wants to move it.

And the list has to be written down in advance, by people with the authority to bind the organisation, in a document that cannot be amended by the executive who finds it inconvenient on a Tuesday. Otherwise, the prohibition exists at the discretion of whoever is under the most pressure, which is to say it does not exist.

What It Would Have Cost

Return to the tablets for a moment, because the arithmetic here is instructive.

If somebody at 7-Eleven had asked the question — should we take a biometric reading of every customer who fills in this survey? The answer would have been no, and it would have taken about four seconds to reach that conclusion. Nobody would have argued. The purpose was to survey hygiene. The alternative was a checkbox.

The cost of asking would have been nothing. The cost of not asking was 1.6 million people’s faces, a regulatory determination, a destruction order, and a permanent entry in the case law of what Australian companies may not do.

That is the shape of this failure almost every time. The prohibition, when finally examined, is obvious. The harm is real. The benefit was trivial. And the reason it happened, anyway, is that the organisation had no place in its process for the obvious question to be asked out loud by someone whose job it was to ask it.

The uses you would be ashamed to defend are rarely the ones anyone argued for. They are the ones nobody was required to question.

The Line, and Who Draws It

Nothing external is going to draw this line. That was the argument of the last essay, and it applies here with full force: there is no Australian regulator who will inspect your use case before you deploy it, and the guardrail that would have required you to assess it was withdrawn.

So the line is yours to draw, and the only question is whether you draw it before the capability arrives or after the harm does. 7-Eleven drew it after. The determination did it for them, at a cost, and the line it drew is now permanent and public.

An organisation serious about this decides, in advance and in writing, what it will not do with AI — surveillance of people who have not meaningfully consented, inference of characteristics people have not disclosed, automated decisions that deny someone something essential with no human being who can be reached, systems designed to exploit the fact that a person is desperate or confused or a child. Then it puts that list somewhere a project cannot route around, and it gives someone the authority to stop anything that crosses it, with the standing to survive doing so.

None of this is difficult. It requires no technology, no new regulation, and no great moral courage. It requires only that somewhere in the process between a vendor’s feature list and 700 stores, there is a step where someone has to ask the question out loud.

Nobody at 7-Eleven asked. That is the whole story. It is also, right now, the story in most organisations deploying AI — where the capability is arriving faster than anyone’s willingness to say that some of it is simply not for us.

References

This piece draws on determinations of the Office of the Australian Information Commissioner and the published record of Australian AI policy.

1. Office of the Australian Information Commissioner. Commissioner initiated investigation into 7-Eleven Stores Pty Ltd (Privacy) [2021] AICmr 50. Determination of Commissioner Angelene Falk, 29 September 2021.

2. Office of the Australian Information Commissioner. Commissioner initiated investigation into Clearview AI, Inc. (Privacy) [2021] AICmr 54. Determination, 14 October 2021.

3. Office of the Australian Information Commissioner. Statement on Clearview AI. Canberra, 21 August 2024.

4. Department of Industry, Science and Resources. National AI Plan. Canberra, December 2025.

Fifteen months later, the government abandoned them.

The National AI Plan, released on 2 December 2025, dropped the mandatory guardrails and the idea of an AI Act. In their place: existing technology-neutral laws, on the theory that privacy, consumer protection, and anti-discrimination statutes already cover whatever AI does; and voluntary guidance, in the form of six essential practices published two months earlier, which organisations are encouraged to adopt. An AI Safety Institute was funded to monitor, test, and advise.

It has no enforcement powers.

What Was on the List

Read the abandoned guardrails closely, because the specifics matter more than the fact of the retreat.

One of the ten was contestability: a requirement that people affected by a high-risk automated decision be able to challenge it. Another was human oversight: a requirement that a person be able to intervene in, and override, what the system did. Another was accountability: a requirement that someone be identifiable and answerable for the system’s outcomes. Another was transparency: that people be told a machine was involved at all.

Those four, had they been law, would have described with some precision the protections that Robodebt’s victims did not have. They could not contest because the burden had been inverted, and the tribunal decisions were never allowed to become precedent. There was a human presence but no human authority. Nobody could be identified as accountable until a royal commission was convened to find out. And the scheme’s logic was opaque to the people it judged.

So the guardrails were, in a real sense, drafted from the wreckage. They were the state’s own account of what had gone wrong, converted into obligations.

And then they were withdrawn.

The protections Australia decided not to mandate are the very protections whose absence led to its worst automated-decision failure. That is not an accident of drafting. It is a decision about who bears the risk.

The Argument for Voluntary

The case for the retreat is not stupid, and it deserves to be stated properly rather than caricatured.

Australian law is technology-neutral, and that is a genuine strength. The Privacy Act does not distinguish between a human and a model mishandling your data. Consumer law does not care whether a person or an algorithm misled you. Anti-discrimination statutes apply to a discriminatory outcome regardless of what produced it. Directors’ duties do not contain an exception for decisions taken by software. On this reading, AI-specific legislation risks building a parallel regime that duplicates what already exists, freezes today’s technology into tomorrow’s statute, and imposes compliance costs on a country that is already a net importer of AI rather than a builder of it.

The Productivity Commission made this case explicitly, calling for a pause on economy-wide AI regulation. Industry made it forcefully. And there is a version of it that is simply true: an organisation that breaks the Privacy Act with a model is still breaking the Privacy Act.

But notice what that argument quietly concedes. It says that existing law will catch up to you afterwards. It says nothing about whether anything will stop you first.

Enforcement After the Fact Is Not Protection

Robodebt was unlawful under existing law. It was unlawful from the first debt notice in 2015, and the legal advice saying so existed in 2014. Technology-neutral law applied to it perfectly, and technology-neutral law did not stop it. The Federal Court eventually found the method unlawful — in 2019, four years and hundreds of thousands of victims later.

That is what existing law enforcement looks like when a machine is running at scale. It is retrospective. It requires someone harmed to litigate, or a regulator to investigate, or a commission to be convened. It arrives after the money has been taken and, in some cases, after the person is dead. The law was not absent. It was slow and downstream, and the machine was fast and upstream.

A statute that catches you in four years is not a guardrail. It is a post-mortem with a fine attached.

This is the gap the mandatory guardrails were designed to close, and it is precisely the gap the National AI Plan reopened. Conformity assessment before deployment, testing before release, a contestation route that exists on day one rather than being litigated into existence on day 1,400 — these are ex ante controls. They act before the harm. Existing law, however elegant and technology-neutral, acts after it.

Australia has decided that ex ante protection for high-risk AI is optional. Not absent. Optional. There is a difference, and the difference is where this essay is going.

The Word That Does the Work

The six essential practices published in October 2025 are, on their merits, good. They cover governance and accountability, risk management, data governance, testing and monitoring, privacy, and continuous improvement. An organisation that implemented them properly would be in a far better position than most. Nothing here is an argument against their content.

The argument is about the adjective. They are voluntary.

Voluntary means an organisation adopts them when it wants to and does not when it does not. It means the practices bind only as far as the enthusiasm of the person implementing them goes, and no further. It means that when a delivery deadline collides with a testing requirement, nothing external decides the conflict. It means the framework is a description of what a responsible organisation would do, offered to organisations that may or may not be responsible.

And it means the guidance has the same enforcement architecture as Enron’s code of ethics, which its board formally voted to waive so that its chief financial officer could run the partnerships that destroyed the company. A rule you can switch off is not a rule. It is a preference with good branding.

What This Actually Means for You

Strip away the policy language, and the position is stark, and it is worth stating without euphemism.

If your organisation deploys a system that decides something consequential about a person — whether they get the loan, the job, the insurance, the tenancy, the benefit — there is no Australian regulator that will inspect that system before it goes live. Nobody will require you to test it for bias. Nobody will require you to explain your decision to the person it affected. Nobody will require you to give that person a route to challenge it. Nobody will require you to name a human being who is accountable for what it does.

You may do all of those things. You are encouraged to. There is excellent guidance available, and it is free.

But if you do not, the machine still runs. It runs until someone who was harmed finds a lawyer, or a journalist, or the resources and stamina to litigate against an institution with vastly more of both. It runs until the harm is large enough and documented enough to summon the retrospective machinery of existing law — which, on the Robodebt evidence, takes years and requires the dead to pile up first.

The person on the other side of your model has no external protection. Whatever protects them is whatever you built.

The Burden Nobody Asked For

This is not a call for regulation, nor a complaint that the government failed to do its job. Reasonable people can hold that a standalone AI Act would have been clumsy, premature, and economically costly for a country in Australia’s position. That argument may even be right.

But the argument has a consequence, and the consequence is not evenly distributed. When the state declines to mandate ex ante protections, it does not eliminate the need for them. It relocates the decision. The question of whether a person can contest an automated decision that ruined their week does not disappear because the guardrail was withdrawn. It simply moves from parliament to a product meeting, where it will be decided by people who balance it against the launch date.

Deregulation does not remove a control. It moves the decision about that control from a legislature to a delivery team, and hopes.

So the burden has landed on the enterprise, and it has landed without ceremony. Every protection the guardrails would have compelled is now a thing your organisation must choose to build, fund, and — this is the hard part — enforce against itself, under commercial pressure, with nobody watching and no penalty for quietly deciding not to.

Most organisations will not build it. They will adopt the principles, publish them, and treat the publication as the work. This is not cynicism; it is the observable pattern of every voluntary code in the history of corporate governance. The organisations that build real controls will be the ones that decide the protection matters independently of whether anyone is going to make them.

Which Leaves the Question

Australia has run an experiment, and the terms are now clear. It has been that existing law, plus good guidance, plus the good faith of the people deploying these systems, is enough. The AI Safety Institute will watch and advise. Nobody will be compelled.

Which means the question for anyone deploying AI that touches human beings is no longer what the regulator will require. There is no answer to that question. The question is what you would build if nobody were going to check — because that is precisely the situation you are in.

It is an uncomfortable question, and it is uncomfortable in a specific way: it is the question a person’s character answers, not their compliance function. There is no filing to submit and no auditor to satisfy. There is only the gap between what you know you should build and what you can be made to build, and the discovery that in this country, for now, those are very different sizes.

The guardrails were withdrawn. The institute has no teeth. And the person your model is about to judge is relying entirely on what you decided to build when you thought no one was looking.

References

This piece draws on the Australian Government’s published AI policy record and reporting on the National AI Plan.

1. Department of Industry, Science and Resources. National AI Plan. Canberra, December 2025.

2. Department of Industry, Science and Resources. Proposals Paper: Introducing Mandatory Guardrails for Safe and Responsible AI in Australia. Canberra, September 2024.

3. National Artificial Intelligence Centre. Guidance for AI Adoption: Foundations and Implementation Practices. Canberra, 21 October 2025.

4. Department of Industry, Science and Resources. Voluntary AI Safety Standard. Canberra, September 2024.

5. Productivity Commission. Interim report on harnessing data and digital technology, August 2025.

6. Royal Commission into the Robodebt Scheme. Report of the Royal Commission into the Robodebt Scheme. Canberra, 7 July 2023.

The system was built. It ran from 2015 to 2019. It issued debt notices to hundreds of thousands of people on welfare payments, and it inverted the burden of proof: the machine asserted the debt, and the person had to disprove it, often by producing payslips from years earlier that they had no means of obtaining. When the Federal Court finally examined the method in 2019, it was unlawful. By then, the Commonwealth had taken at least $751 million from people on the basis of debts it had no legal power to raise.

The Royal Commission that eventually examined the scheme described it as a crude and cruel mechanism, neither fair nor legal, and said that it made many people feel like criminals. Mothers gave evidence about sons who had taken their own lives after receiving debts they could not disprove.

The Knowledge Was Never Missing

The temptation is to file Robodebt under cruelty or incompetence and move on. Both readings are comfortable, and both are wrong, because they suggest the problem was that nobody knew.

Everybody knew. The legal advice existed in 2014, before a single debt was raised. The Commonwealth Ombudsman investigated in 2017. The Senate inquired twice. The Administrative Appeals Tribunal struck down income averaging in decision after decision at the first tier, and the department did not appeal any of them, because an appeal would have produced a binding precedent it did not want. Each adverse ruling was absorbed in silence, and the machine kept running. Welfare recipients wrote, called, and pleaded. Community lawyers documented it. Journalists reported it.

Four years of knowing, and the scheme did not stop until a court ordered it to.

The information was never the missing ingredient. What was missing was any mechanism by which knowing became stopping.

Everyone Was Responsible, So Nobody Was

Ask who was accountable for Robodebt, and the answers arrive in a circle.

The ministers acted on departmental advice. The department acted with ministerial authority. The public servants who had doubts raised them through channels, and the channels absorbed them. The officers who sent the letters were executing a process they had not designed and could not override. And the system itself — the actual calculation that averaged the income and generated the debt — had no author in any meaningful sense. It was a policy expressed as arithmetic, and arithmetic cannot be culpable.

The Royal Commission found that a departmental secretary had been made aware of the scheme’s illegality and failed to act. It found a minister had allowed the cabinet to be misled. But those findings had to be constructed, painstakingly, across three volumes and hundreds of pages, by a body with royal powers of compulsion, four years after the scheme had ended and eight years after the first unlawful debt was raised.

That is what it took to answer a question that should have been answerable on the first day: who is responsible for what this machine does to people?

An accountability that must be excavated after the harm, by an inquiry with the power to compel testimony, is not accountability. It is archaeology.

The Tool That Isn’t

Underneath the circle of deferrals sits a single comfortable idea, and almost every organisation deploying AI today lives inside it: the system is a tool; tools have users; and users are accountable.

This is how we think about hammers, and it is unimpeachable right up to the moment the tool stops executing a decision and starts making one. Then a gap opens in the middle of the chain that runs from human intention to human consequence, and nobody has been assigned to close it.

Consider what user accountability actually meant inside Robodebt. The user was a Centrelink officer who did not design the averaging method, could not inspect it, had a queue and a handling-time target, and whose entire practical discretion consisted of forwarding what the system had produced. To call that person accountable is not a governance position. It is a place to put the blame when it eventually arrives.

And notice where responsibility goes when the fiction collapses. It does not transfer to the machine, which cannot be sanctioned, struck off, or compensate anyone. It does not settle on the executive, who never saw an individual decision. It does not rest with the engineer, who built to specification. It does not attach to the policy officer, who wrote a rule, not a system.

It simply becomes unlocated. That is the accountability gap: not an absence of responsible people, but the absence of any answer, settled in advance and in writing, to the question of which one.

Delegating a decision is not delegating responsibility for it. Most organisations have done the first and quietly assumed the second came along.

A Human Was in the Loop the Whole Time

The standard reassurance is that a human remains in the loop. The phrase has done more damage than almost any other in this field, because it describes a position and implies an authority, and those are not the same thing.

A human in the loop, in most real deployments, is a person who reviews what the system surfaced, under time pressure, with a queue behind them, in an environment where agreeing with the machine is fast and costless and disagreeing is slow and demands justification. Every incentive arranged around that person produces assent. They are present. They are not, in any operative sense, deciding.

Robodebt had humans throughout — officers, managers, secretaries, ministers, an ombudsman, a tribunal, two Senate committees. What it did not have was a single point at which a named human with real authority was required to examine the method and say yes or no, in writing, before the machine reached a citizen. Presence was everywhere. Authority was nowhere.

The Objection, and Why It Fails

There is an objection to all of this, and it deserves to be met at its strongest rather than its weakest.

It runs like this. Robodebt was not an AI failure. Its algorithm was arithmetic — a division, performed at scale. No machine learning, no neural network, nothing that could not be done in a spreadsheet. What went wrong was political culture, ministerial ambition, and a public service that had learned not to say no; the Commission itself named venality, incompetence and cowardice. To recast that as an algorithmic accountability problem is to let humans off the hook by blaming the technology’s shape. Large organisations have always diffused responsibility. Committees have always been where accountability goes to die. There is nothing new here.

Every part of that is true, and it strengthens the case rather than weakening it. If a division sum could open an accountability vacuum that took a royal commission four years to excavate, then nobody should want to argue that systems which are more capable, more opaque, and more autonomous will be easier to hold to account. Robodebt is not the exception that proves AI will be fine. It is the floor.

The technology changes one thing, and it changes it decisively. Robodebt ran for four years and reached hundreds of thousands of people before the courts caught it, and it was slow enough and paper-based enough to leave a trail. That trail is the only reason the scheme has a name. A modern system does the same amount of harm before a quarterly report prints, and it leaves a log that nobody reads, instead of a letter that somebody keeps.

What Closing It Requires

If the gap is the absence of a pre-assigned answer, it cannot be closed by better values, stronger ethics training, or a more sincere statement of principles. It is closed by deciding three things before the system is permitted to act, and writing them down where they can be produced later.

Who is accountable for this system’s outcomes by name? Not a committee, not a function, not the business. A person who can be asked afterwards why they permitted this, and who knew in advance that they would be asked.

What that person had to see before the machine was allowed to decide. The population it will touch, the errors it will make, the harm those errors will cause, and the route by which a person on the receiving end can push back. A signature on a document nobody read is the same fiction with a name attached to it.

What happens when it goes wrong, mechanically rather than aspirationally? Not that concerns will be escalated, which is precisely what Robodebt had, in abundance, for four years. But that’s the thing that stops. That there is a point at which the machine’s authority ends and it cannot proceed, and that point is enforced by architecture rather than by somebody’s willingness to make a career-limiting phone call.

None of this is exotic, and none of it is new. It is what every other domain of consequential machinery worked out long ago. An aircraft does not fly because its designers had good intentions; a named person certifies it, and that person is prosecuted if the certification was a lie. A drug does not reach the market because the chemistry is elegant. The pattern is old and well understood. It has simply not yet been applied to the systems now deciding people’s money, housing, employment, and liberty.

The Question Underneath All the Others

Fairness, explainability, contestability, the provenance of the data — every other question in responsible AI sits downstream of this one, because each is finally an answer to the question of who has to answer for it. Where the answer is nobody, fairness becomes a metric someone tunes until it passes, an explanation becomes a document nobody reads, and a right to contest becomes a form nobody processes.

That is the argument this series will make, and it will take each of those in turn. Why nothing external will stop the next Robodebt, and what that leaves you holding. The uses no organisation should make, and the question that never gets asked before they are made. The fairness threshold somebody chose without telling you. The explanation that satisfies a regulator and tells the person you refused nothing at all. The human review that changes no outcome. The training data, whose origin nobody can state, quietly voids everything built on top of it. And finally, why organisations that agree with every word of this still never build any of it.

Each is a control that fails on its own terms. All of them fail together, and for the same reason, when there is no one whose job it is to stop.

Robodebt had policies. It had oversight bodies, an ombudsman, a tribunal, two Senate inquiries, and a department full of people who knew. What it did not have was a single person who would be held to it, named before the first letter went out, who could not get out of the way.

So the accountability was manufactured afterwards, at enormous public cost, by a commission with the power to compel sworn testimony — which is the most expensive and least effective method of governance ever devised, and the one we currently rely on.

$751 million was taken from people who owed nothing. Some of them did not survive it. And for four years, while everybody knew, there was nobody whose job it was to stop.

References

This piece draws on the public record of the Royal Commission into the Robodebt Scheme, Federal Court judgments, and reports of the Commonwealth Ombudsman and Senate committees.

1. Royal Commission into the Robodebt Scheme. Report of the Royal Commission into the Robodebt Scheme. Commissioner Catherine Holmes AC SC. Canberra, 7 July 2023.

2. Federal Court of Australia. Amato v Commonwealth. Consent judgment finding the income-averaging method unlawful, November 2019.

3. Commonwealth Ombudsman. Centrelink’s Automated Debt Raising and Recovery System. Report No. 02/2017, April 2017.

4. Senate Community Affairs References Committee. Inquiries into the Better Management of the Social Welfare System initiative. Canberra, 2017 and 2022.

5. Federal Court of Australia. Prygodicz v Commonwealth of Australia (No 2) [2021] FCA 634. Settlement approval judgment, June 2021.