Designed to Blame
What we call human error, and who is left holding it
❝Serious accidents are frequently blamed on ‘human error.’ Yet careful analysis of such situations shows that the design or installation of the equipment has contributed significantly to the problems.❞
— Don Norman1
The Letter
A letter arrives at a house in the Netherlands. It is from the tax office, and it tells the family that the childcare money they have had for years was never theirs to take, and that they must now repay all of it, at once, tens of thousands of euros. There is no number that will explain how the decision was made, because it was not made by anyone they can reach. Somewhere upstream, they had been scored for fraud risk, and nationality had been used as one of the risk factors in that score, which is how a family came to be treated as a suspect for reasons that had little to do with anything they had done.2 By the time the Dutch state admitted what it had built, tens of thousands of families had been put through some version of that morning, and 1,115 children had been taken from their parents and placed in care.3
Nobody in this story woke up wanting to ruin a family. That is the part worth staying with, because it is the part that lets it happen again.
Up the Chain
The caseworker enforcing the demand was only reading a flag. The flag had come from a model, and the model had been built to catch fraud by people who were nowhere near the kitchen when the letter landed on the mat. Each link in that chain could point back to the one before it and say, with a straight face and some truth, that it had only done its own small part. Follow it all the way up, and you reach the place where the last of these essays ended, an empty chair where the decision was supposed to have a face. The hand had been designed out of this one, too.
What is different here is that the empty chair did not stay empty for long. Something had to fill it, because a family was being held responsible, and responsibility is not allowed to point at anybody. So it pointed down, at them.
The Useful Word
This is the move the profession has the least stomach for. When a system built by many hands falls hard on the person at the end of it, we keep a phrase ready: human error. The user clicked the wrong thing. The customer should have read the small print before agreeing, and the parent, surely, should have asked about it sooner. Don Norman, who spent a career looking harder at ordinary things than the rest of us manage, kept pointing out that the accidents we pin on human error usually trace back to the way the thing was designed, and that the honest name for most of it would put the fault back on the system rather than the person standing in front of it.
We have mostly said no. Human error is too useful a phrase to give up. It ends the matter at the cheapest point, the person standing closest to the damage, and leaves everyone behind them exactly where they were.
Downhill
Once you have the shape, it turns up wherever the distance runs. A pulse oximeter reads normal on darker skin, and the patient is logged as less ill than she is; her decline is filed under her own body rather than the instrument that could not see it.4 With the exam grades one summer, a formula the students never saw reached down and overruled the marks their teachers had given them, and the young people left holding the lower result were the ones told to accept it, until enough of them refused in the street that the whole thing was pulled.5 The customer who cannot find the way out of a subscription is treated as confused, even though the way out has been routed through so many screens that regulators have given the obstruction a name and gone after the firm that built it.6
Each of these hands a real person a fault that was assigned to them rather than earned. And the assignment keeps moving the same way, toward whoever has the least standing to send it back.
Who It Finds
Because the blame knows how to choose. It does not fall at random. The direction is always downward, well clear of the people who set the thing in motion, settling instead on the person least able to throw it off, the one without the lawyer, or the standing that makes an institution hesitate before doubting them. The same family that gets scored as a risk is the family least equipped to contest the score, and that overlap is the cheapest way for the system to run. The disappearance of the maker and the arrival of a culprit are one and the same event, and the culprit is chosen for being easy to convict. What looks, from inside the profession, like a clean and neutral process is, seen from underneath, an allocation of blame that, every time, falls on the people with the least left to push back with.
What Doesn’t Come Back
Some of it gets put right in the end. Convictions get quashed. The money is returned, with interest even, and an apology is read out by someone senior enough to be comfortable reading it. What does not come back is the stretch of time in which the person believed it, the months or the years of going looking for the flaw in themselves, because everything around them was so smooth and so certain that the error had to be the human’s. The maker is still not in the room. The person at the bottom is still holding the thing that was never theirs to hold.
We called it human error, and we moved on.
NK
☉
Thanks for reading Designverse. If this said something true, pass it to someone who makes things.
Don Norman, The Design of Everyday Things (rev. ed., Basic Books, 2013). Don Norman is an American researcher, professor, and author widely revered as the “Father of User Experience (UX)”. He famously became the first person to use “User Experience” in a job title, serving as the UX Architect at Apple in 1993. Norman is globally renowned for pioneering user-centred design principles and co-founding the influential usability consulting firm Nielsen Norman Group (NN/g) alongside Jakob Nielsen.
The Dutch childcare benefits scandal, the toeslagenaffaire. The tax administration used nationality as one of the risk factors in an automated fraud-detection system, and Amnesty International characterised the outcome as discrimination and racial profiling. Amnesty International, Xenophobic Machines: Discrimination Through Unregulated Use of Algorithms in the Dutch Childcare Benefits Scandal (October 2021).
On children removed from affected families, a government-supported count records 1,115 children taken into care between 2015 and 2020; a later Dutch government figure identifies 1,819 children of 1,168 parents over a wider period. The draft uses the 2015 to 2020 count. The wider scale, tens of thousands of families wrongly pursued, is supported in inquiry reporting and later ministry acknowledgements and is presented as reported.
Michael W. Sjoding et al., “Racial Bias in Pulse Oximetry Measurement,” New England Journal of Medicine, N Engl J Med 2020;383:2477–2478, DOI 10.1056/NEJMc2029240 (16 December 2020). The device failed to detect dangerously low blood oxygen, occult hypoxemia, far more often in Black patients than in white patients.
England’s 2020 A-level grading. With exams cancelled during the pandemic, Ofqual’s standardisation algorithm adjusted teacher-assessed grades, and after public protest, the government reverted to teacher assessment. Ofqual’s own 2020 analysis stated that it found no evidence of systemic bias against disadvantaged pupils, so the essay rests only on the uncontested facts that a formula the students never saw overrode their teachers’ grades and was then withdrawn. The Guardian, “A-level and GCSE results in England to be based on teacher assessments in U-turn.”
FTC v. Amazon.com, Inc. (2023), concerning a Prime cancellation process the company internally nicknamed “Iliad,” described by regulators as deliberately onerous. The FTC has also taken action against Epic Games (2022) for deceptive interface design. FTC, “Amazon.com, Inc. (ROSCA), FTC v.”