Police use of AI technology is ‘outpacing scrutiny and regulation’, warns report

The Lords Justice and Home Affairs Committee warned new technologies are developing in “a new Wild West” at a pace that public awareness, government and legislation have not kept up with.

While it acknowledged the advantages AI technology can bring to the justice system, the committee said it could have “serious implications for a person’s human rights and civil liberties”.

Published on Wednesday (March 30), the report, Technology rules? The advent of new technologies in the justice system, says: “Algorithms are being used to improve crime detection, aid the security categorisation of prisoners, streamline entry clearance processes at our borders and generate new insights that feed into the entire criminal justice pipeline.”

But it said “informed scrutiny was therefore essential to ensure that any new tools deployed in this sphere are safe, necessary, proportionate, and effective”.

“This scrutiny is not happening,” it added.

“Instead, we uncovered a landscape, a new Wild West, in which new technologies are developing at a pace that public awareness, government and legislation have not kept up with.”

This lack of oversight meant “users are in effect making it up as they go along”, it said.

The committee was also concerned that police forces and public bodies were commissioning tools in a “worryingly opaque” AI market due to the companies’ insistence on commercial confidentiality – despite the fact that many of these systems will be “harvesting, and relying on, data from the general public”.

The report noted that “thanks to its ability to identify patterns within data, AI is increasingly being used in ‘predictive policing’ – forecasting crime before it happens”.

But while AI “therefore offers a huge opportunity to better prevent crime” there is also a risk it could exacerbate discrimination, the committee cautioned.

It said it heard repeated concerns about the dangers of human bias contained in the original data being reflected, and further embedded, in decisions made by algorithms.

“Without transparency, there can not only be no scrutiny, but no accountability for when things go wrong,” said the committee.

It wants to see the establishment of a mandatory register of algorithms used by the police and in the justice system.

It also recommends the introduction of a “duty of candour” on the police to ensure full transparency over their use of AI given its “potential impact on people’s lives, particularly those in marginalised communities”.

To clarify governance and guarantee the integrity of the technologies being deployed, the committee is calling for a national body to be established to set strict scientific, validity, and quality standards and to certify new technological solutions against those standards.

“In relation to the police, individual forces must have the freedom to engage the solutions that will address the problems particular to their area, but no tool should be introduced without receiving ‘kitemark’ certification first,” says the committee.

To secure the good use and close monitoring of new technologies, it also recommends there should be mandatory training for officers and officials on the use of the tools themselves as well as general training on the legislative context, the possibility of bias and the need for cautious interpretation of the outputs. Local ethics committees should also be developed within police forces.

The committee argues that, as the use of new technologies is becoming routine, these proposed reforms will “ensure that we maximise their potential while minimising the associated risks”.

“They would reverse the status quo in which a culture of deference towards new technologies means the benefits are being minimised, and the risks maximised,” it added.

The committee said it was clear the system “needs urgent streamlining” and reforms to governance should be supported by a strong legal framework.

“As it stands, users are in effect making it up as they go along,” it says.

“Yet without sufficient safeguards, supervision, and caution, advanced technologies may have a chilling effect on a range of human rights, undermine the fairness of trials, weaken the rule of law, further exacerbate existing inequalities, and fail to produce the promised effectiveness and efficiency gains.”

While the committee acknowledged the “good intentions” of users, it said good intentions were not enough and legislation should be introduced to establish clear principles applicable to the use of new technologies and how these principles must be applied in practice.

Baroness Hamwee, chair of the Justice and Home Affairs Committee, said: “What would it be like to be convicted and imprisoned on the basis of AI which you don’t understand and which you can’t challenge?

“Without proper safeguards, advanced technologies may affect human rights, undermine the fairness of trials, worsen inequalities and weaken the rule of law. The tools available must be fit for purpose, and not be used unchecked.

“We had a strong impression that these new tools are being used without questioning whether they always produce a justified outcome. Is ‘the computer’ always right? It was different technology, but look at what happened to hundreds of Post Office managers.

“Government must take control. Legislation to establish clear principles would provide a basis for more detailed regulation. A ‘kitemark’ to certify quality and a register of algorithms used in relevant tools would give confidence to everyone – users and citizens.”

Baroness Hamwee added: “We welcome the advantages AI can bring to our justice system, but not if there is no adequate oversight. Humans must be the ultimate decision makers, knowing how to question the tools they are using and how to challenge their outcome.”

In the EU, more than 40 civil organisations are calling on the Council of the European Union, the European Parliament and all EU member state governments to prohibit AI predictive and profiling AI systems in law enforcement and criminal justice in the proposed Artificial Intelligence Act (AIA).

They argue that AI systems in law enforcement, particularly the use of predictive and profiling AI systems, “disproportionately target the most marginalised in society, infringe on liberty and fair trial rights, and reinforce structural discrimination”.

“The only way to protect people from these harms and other fundamental rights infringements is to prohibit their use,” says Griff Ferris, legal and policy officer at Fair Trials

And Sarah Chander, senior policy adviser at EDRi (European Digital Rights) says AI for predictive policing is “reinforcing racial profiling and compromising the rights of over-policed communities”, adding: “The EU’s AI Act must prevent this with a prohibition on predictive policing systems.”

The College of Policing recently published new national guidance on live facial recognition technology guidance, which includes “a clear legal and ethical framework for its use”.

In particular, the authorised professional practice (APP) makes it clear that “live facial recognition technology should be used in a responsible, transparent, fair and ethical way and only when other, less intrusive methods would not achieve the same results”.

It also states that:

• All use of the technology has to be targeted, based on intelligence and have a set time for use to start and end;
• Images placed on the predetermined database must meet a proportionality and necessity criteria and be reviewed before every deployment;
• Police forces should give notice about the use of live facial recognition, making use of social media and their website, unless there is a critical threat that makes this impractical; and
• Chief officers should involve their elected police and crime commissioner to provide oversight.

The new APP comes after consultation with the public, police, regulatory bodies and campaign groups. It was developed in the light of the Bridges v South Wales Police Appeal Court judgment (2020) and addresses the issues identified in the judgment.