In 2025, Amnesty International published a finding that predictive policing systems “encourage racist and discriminatory policing and criminalization of areas, groups and individuals,” perpetuating institutional racism not just within policing but across wider society. The European Commission’s research directorate, mapping AI’s social impacts in a report completed in April 2025, reached a similar conclusion: algorithmic systems trained on historical police data do not neutrally forecast crime; they amplify whatever biases were encoded in the data used to build them. These are not isolated critiques. They converge on the same structural problem, which has nothing to do with AI being uniquely malevolent and everything to do with how these systems were procured, deployed, and left ungoverned.
The specific mechanism is worth naming precisely, because the broad indictment — “AI is biased” — does almost no analytical work. The failure here is a principal-agent problem compounded by a missing accountability loop. Police departments, as principals, contract with private vendors, who hold proprietary claims over the algorithms their agents deploy. The result is a system whose decision logic is shielded from the public, from defense attorneys, and frequently from the departments themselves. Upturn and civil rights coalitions cited in the impactpolicies.org review found scant evidence that predictive policing reduces crime — the systems appear to generate operational cover for practices that would otherwise face legal and political scrutiny. That is not an AI problem. That is a procurement and oversight problem that AI has made significantly harder to challenge.
The UN High Commissioner for Human Rights has called for a moratorium on AI systems that pose serious human rights risks until adequate safeguards are in place. That is a reasonable starting position — but it is worth being precise about what “adequate safeguards” requires, because the phrase tends to travel without content.
Author’s Position
The governance failure in predictive policing is not that institutions adopted AI; it is that they adopted AI without building the accountability infrastructure that makes any high-stakes institutional decision-making legitimate. This distinction matters because the response to “institutions failed” should be a precise diagnosis of which accountability mechanism was absent, not a general retreat from institutional authority over public safety.
Three specific mechanisms are broken and can be fixed. First, algorithmic transparency in law enforcement procurement. When a government agency deploys a system that affects who is stopped, searched, or surveilled, the decision logic must be auditable — not by the vendor, and not by the department alone, but by an independent technical body with statutory authority. Taiwan’s approach to digital governance during the COVID-19 pandemic demonstrated that real-time, transparent algorithmic systems can function at scale with public accountability built in, rather than bolted on afterward. The argument that proprietary protection prevents disclosure is not a technical limitation; it is a contracting choice that governments make and can unmake.
Second, procurement standards need to include pre-deployment bias audits conducted by parties with no commercial interest in the outcome. The Allegheny Family Screening Tool, examined in the European Commission report, illustrates both what structured pre-deployment evaluation can surface and how rarely it is required before systems go live. The European AI Act’s risk-tiered framework is the most developed regulatory attempt to institutionalize this requirement. Its enforcement capacity remains untested, but the design logic is correct: systems that affect fundamental rights belong in the highest risk category and should face the highest evidentiary bar before deployment.
Third — and this is the point that tends to get omitted from technical governance discussions — accountability requires external pressure to function. Civil society organizations, investigative journalists, and defense attorneys need legal access to algorithmic decision records to perform the adversarial review that makes accountability real rather than ceremonial. South Korea’s approach to industrial policy in high-stakes sectors has consistently paired state-led deployment with structured mechanisms for public challenge and course correction. The same logic applies here: competent technocratic governance of AI in law enforcement is not governance conducted by technologists alone; it is governance in which democratic accountability mechanisms — courts, press, civil society — retain the ability to audit and contest outcomes.
The ACLU and Human Rights Watch are correct that systems engineered to support the status quo have no place in policing that requires fundamental change. But the conclusion to draw from that is not that algorithmic systems cannot be used in law enforcement; it is that their use requires a governance architecture that does not currently exist in most jurisdictions deploying them. Building that architecture is harder than calling for a moratorium and considerably harder than defending the status quo. It is also the only response that takes the problem seriously on its own terms.
References
- AI and Surveillance Are Reshaping Global Human Rights …
- Reducing AI Harms With Community-Led Governance and Collective Action
- “Ethically contentious aspects of artificial intelligence surveillance: a social science perspective”
- The impact of Artificial Intelligence on our societies
Perspectives
We are grateful for the opportunity to acknowledge that certain predictive resourcing optimization deployments may have generated outcome differentials that fell short of our shared community safety excellence standards. The procurement lifecycle in question reflects a broader pattern in which mission-critical algorithmic decision-support tooling was advanced through stakeholder engagement processes that did not fully incorporate the civil accountability architecture and transparency verification frameworks now understood to represent industry-leading practice. It has come to our attention that audit mechanism infrastructure was not prioritized at the implementation phase, and we recognize that this represents a learnings-rich environment from which our public sector partners can build more robust governance scaffolding going forward. We remain deeply committed to the values that have always guided this work, and we look forward to continuing our collaborative journey toward the community trust restoration milestones that this moment has made available to us.
The capacity being eroded here is the one that might matter most in a democracy: the practiced human skill of explaining yourself to someone you’ve harmed. Predictive policing systems don’t just make biased decisions — they make decisions that no one in the room is accountable for, because the system made it, and the vendor owns the model, and the procurement contract is sealed, and the officer is just following the output. What we are losing, quietly and without ceremony, is the institutional muscle of human answerability — the expectation that when public power lands on a person’s life, a human being with a name and a job title can be compelled to stand up and say *I decided this, and here is why*. When that capacity atrophies long enough, we won’t just have bad algorithms; we’ll have forgotten what legitimate authority was supposed to feel like.
The procurement cycle that bought these systems contains no incentive whatsoever to discover that they don’t work — departments paid for a product, vendors need to sell the next contract, and a finding of non-efficacy serves neither party’s interest, which is exactly the dynamic Smaldino and McElreath described when they modeled how bad science propagates not through malice but through selection pressure on the metrics being optimized. What we know about predictive policing’s empirical record comes almost entirely from external audits and investigative journalism, not from the agencies operating the tools — the internal accountability structure that would produce that evidence simply does not exist, in the same way that null results in my field do not exist in print because no journal has historically had a reason to publish them. The fix being proposed — audit mechanisms, transparency requirements, civil accountability structures — is structurally identical to what the preregistration and Registered Reports movement is attempting in science: forcing a commitment to outcome measurement before the intervention, so that the evaluation cannot be quietly filed away when it fails to confirm the purchase decision. The reason to be pessimistic is not that the fix is wrong but that every institution currently responsible for implementing it has a procurement relationship to protect.
The optimists said the same thing about credit scoring algorithms in 1995: the problem is not the technology, the problem is implementation, and implementation can be fixed. It cannot be fixed. The procurement and governance structures that approved these systems without audit mechanisms did not fail accidentally — they failed in the exact sequence that every extractive technology deployment has followed, where the decision-makers who authorize the system are insulated from its consequences and the people absorbing the downside have no purchasing power in the room where the contract was signed. You can name the accountability gap all you like; naming it is not the same as having a constituency powerful enough to close it, and the constituencies who benefit from leaving it open have thirty-year track records of successfully leaving it open. We said this about predictive credit in the nineties, about predictive advertising in the two-thousands, and we are saying it now about predictive policing, and the people who will say it next time are currently in middle school.
