Who answers for the decision
It is a way of ending a conversation, and it works, which is why it is used. The question that reopens the conversation is the one to carry out of this level: who accepted the output. A system proposes; somewhere a person or an institution adopted the proposal, and that is where the account of the decision lives.
Test it against a case where a full audit exists. When public allegations circulated that a credit card's algorithm was giving women lower limits than their husbands, New York's financial regulator examined the underwriting data for nearly 400,000 applicants using regression analysis. It found no evidence of unlawful intentional discrimination and no evidence of a disparate impact fair lending violation (New York Department of Financial Services, 2021).
That looks like a vindication and the report is more interesting than that. The regulator also wrote that even when credit scoring is done in compliance with the law, it can reflect and perpetuate societal inequality, that the lack of transparency to the complainants seemed to produce confusion that could have been mitigated, and this, which is the sentence to remember: federal law mandates that lenders explain only credit denials to applicants, not the reasons for the amount and terms of credit granted.
So the couple who went public were not entitled to an explanation of their limits. Not because a machine was involved, but because no such right existed. The algorithm was the visible thing to blame in a situation where the real absence was a legal one.
The reverse case is where automation removes an account that used to exist. At least fourteen people in the United States are known to have been wrongfully arrested after police relied on a face recognition match, most of them Black; one person spent six months in jail (ACLU, April 2026). Fourteen is a floor rather than a count, because it depends on somebody discovering that the technology was used, and most jurisdictions do not record it in charging documents. There is no official national tally.
This is why the property in Lesson 0.1 matters practically. When a decision comes out of a learned system, the questions that produce an answer are: what was the system's output, what did a person do with it, and what would have happened if the output had been different.
In the government test lab's summary of demographic differentials, for one highly accurate algorithm at a nominal false match rate of one in about 33,000, false match rates ran from roughly one in 26,000 for Polish men aged 35 to 50, to roughly one in 35 for Nigerian women aged 60 and over (NIST, NISTIR 8429, 2022).
That is a spread of about 750 times inside one algorithm, and one of the most accurate the laboratory tested, with older Nigerian women at the wrong end of it.
The report is careful about the cause: false positive differentials arise from under-representation of a demographic in the training data. It is not a property of faces. It is a property of what was collected.
Two provisions are worth knowing by number, because they change what you can ask for.
Under Article 18(1)(f) of Nigeria's General Application and Implementation Directive 2025, consent is required before a data controller makes a decision based solely on automated processing which produces legal effects concerning or significantly affecting the data subject. Article 28(3)(b) makes a data privacy impact assessment mandatory for automated decision-making with legal or similar significant effects.
The words doing the work are solely and significantly. A system that scores you and a person who then decides is not solely automated, which is why so many processes are designed with a person in the loop who approves nearly everything. Whether that person is exercising judgement or providing legal cover is a question about their workload and their incentives, not about the technology.
This course reports those article numbers as moderately sourced. The authoritative gazetted text could not be opened directly; the numbering was consistent across two independent sources. Before quoting a section number in a formal complaint, open the primary document.
The general point survives whatever the numbering turns out to be. Ask who accepted the output, not what produced it.
A staff member tells a customer that her application was declined by the system and there is nothing anyone can do. What is the accurate description of what has happened?
The regulator's report on the credit card allegations found no fair lending violation. What is the most useful thing to take from it?
Notes are kept with your account, alongside your progress and your gate claims. The lesson itself is readable without one.
