Thönnes, Christian

Vavoula, Niovi

English

<intR>²Dok

Automated predictive threat detectionafter Ligue des Droits HumainsChristian Thönnes 2023-05-08T11:52:10On 21 June 2022, the Court of Justice of the European Union (CJEU) released itsjudgment regarding the compatibility of the EU Directive on Passenger Name RecordData (PNR Directive) with the rights to privacy and personal data protection. Liguedes droits humains has already qualified as a landmark decision, where the Courthad the opportunity, among other aspects, to provide comprehensive guidelineson how large-scale predictive policing should take place. In so doing, the Courtfollowed in the footsteps of La Quadrature du Net, and Opinion 1/15, which tackle arelentlessly growing kit of big-data-based security instruments.In the past few years, various security law instruments (both adopted and proposed)have required automated predictive threat detection instruments, meaning thatthey automatically sift through massive amounts of data in order to predict potentialthreats to public security. Given this context, Ligue des droits humains could be usedas an inspiration for their legal assessment.Against this backdrop, this two-part contribution aims to first provide an outline ofthe CJEU’s findings in Ligue des droits humains on automated predictive threatdetection (Part I), and then (in Part II, to be published on 12 May 2023) to analyseits implications for two other legal instruments relying on automated predictivethreat detection: the Regulation establishing a European Travel Information andAuthorisation System (ETIAS) and the Commission’s proposal for a Regulation oncombating online child sexual abuse material (CSAM). Here, we argue that, whilethere remains room for elaboration, Ligue des droits contains a plethora of relevantstandards for future security instruments.Automated predictive threat detection in Ligue desDroits HumainsThe CJEU imposed significant limits on the creation and use of algorithms and otherforms of modern technology for security purposes. Ligue des droits humains gavethe Court the opportunity to do so because the PNR Directive obliges designatednational security authorities, so-called Passenger Information Units (PIUs), toautomatically process PNR data by comparison, not only against pre-existingdatabases (Art. 6 (3)(a), but also “pre-determined criteria”. The latter are algorithmswhich, in the Commission’s words, contain “search criteria, based on the past andongoing criminal investigations and intelligence, which allow to filter out passengerswhich corresponds to certain abstract profiles […]” (see here, page 11 footnote 36).According to the Commission, pre-determined criteria serve to “identify persons- 1 -involved in criminal or terrorist activities who are, as of yet, not known to the lawenforcement authorities.” (see here, page 24).First, the judgment restricted the “use of artificial intelligence technology in self-learning systems (‘machine learning’)”, by prohibiting systems that are “capableof modifying without human intervention or review the assessment process and,in particular, the assessment criteria on which the result of the application ofthat process is based as well as the weighting of those criteria” (para 194). Aspreviously noted, the exact scope of that prohibition is debatable. What is clear,however, is that the Court insists on the necessity of meaningful human interventionin predictive policing systems – a general principle already enshrined in Art. 11of the Law Enforcement Directive. Central for its assessments was the “opacitywhich characterises the way in which artificial intelligence technology works” (para.195), which may “deprive the data subjects […] of their right to an effective judicialremedy” (para 195).The Court further highlighted the risk of discrimination. While the PNR Directivealready acknowledged such risks in its Art. 6(4), the Court nowalso emphasisedthat that provision covers both direct and indirect discrimination (para 197). Thisis crucial because pre-established criteria may be based on seemingly innocuouspersonal data, which may, however, be proxies of prohibited characteristics. Forexample, a person’s address may also be used as a proxy for religion, race or ethnicorigin. Algorithms must be “targeted, proportionate and specific” (para 198) and thusnon-discriminatory – a finding with wider implications in the context of migration,where non-discrimination must be (but is not) embedded in the discretionarydecision-making process. Used technologies will also have to comply with a set ofadditional quality standards: They will have to incorporate “incriminating” as wellas “exonerating’ circumstances” (para 200), thus bolstering their reliability andreducing false-positive rates. The Court stated that high false-positive rates, whichare present in Member States’ statistics, may undermine a system’s suitability andproportionality (see para 123). Thus, the CJEU stressed the necessity of regularreviews of the pre-determined criteria’s strict necessity (para 201). In addition,the Court underscored that the Data Protection Officer and national supervisoryauthorities must be equipped with robust rights of access to the content of pre-determined criteria (para 212).Whilst acknowledging “the fairly substantial number of ‘false positives’”, the CJEUstressed that “the appropriateness of the system […] essentially depends on theproper functioning of the subsequent verification of the results […] by non-automatedmeans” (para 124). For that purpose, the Court determined that Member Statesmust “lay down clear and precise rules capable of providing guidance” for the review(para 205), which is meant to prevent both discriminatory results, and false matchesto be transferred to the competent authorities, thus subjecting passengers to falsesuspicions of being involved in terrorist offences or serious crimes. It also aimedat reliable documentation and self-monitoring (para 207), as well as guaranteeinguniform administrative practices across PIUs in different Member States thatobserves the principle of non-discrimination. The results of individual human reviewsmust take preference over those of automated processing (para 208).- 2 -Finally, the judgment bolstered the right to an effective judicial remedy, as enshrinedin Art. 47 of the Charter of Fundamental Rights of the European Union: “Thecompetent authorities must ensure that the person concerned […] is able tounderstand how those criteria and those programs work”, so that they can “decidewith full knowledge of the relevant facts whether or not to exercise [their] right to thejudicial redress”, pursuant to Art. 13 of the PNR Directive (para 210). This seems toimply notification requirements in cases of verified positive matches which currentlyneither the PNR Directive nor most national transposition laws expressly contain.Room for elaborationWhereas the Court established an abundance of procedural safeguards to reign inthe potential excesses of automated predictive threat detection, Ligue des droitshumains also left a lot of open questions.First, while the Court rightly flagged false-positives as a potential hurdle to asystem’s proportionality, it did not clarify at what point a system just produces toomany of them, thus rendering the system disproportionate. Although this point wasraised in the oral hearing, the Court also never addressed the base rate fallacyundergirding the PNR system. The PNR Directive seeks to identify a very smallnumber of potential terrorists and serious offenders within the general populationof hundreds of millions of annual flight passengers. It therefore, like some otherpredictive policing systems, compels security authorities to look for the proverbialneedle in the haystack. This can result in systemic flaws which make extremely highfalse positive rates a mathematical near-certainty. It remains to be seen whethermere procedural safeguards can succeed in saving the PNR system’s suitability aslong as its underlying base rate fallacy remains unaddressed.Second, it remains unclear what purpose and form human interventions in thePNR system have to take, in particular, how humans are supposed to meaningfullyengage with the PNR system’s automated outputs. The Court delegated theformulation of “clear and precise rules” for human review to Member States (para205) without providing them with much guidance. That PIU officials will be capableof meaningfully engaging with the PNR system’s automated outputs seems doubtfulwhen, for the foreseeable future, they will be confronted with thousands of falsematches.Third, whereas the Court’s insistence on substantive human review is extremelyimportant to prevent automation bias (an often-observed over-reliance onautomatically generated recommendations, see here), there also remainquestions regarding how human review is supposed to prevent direct and indirectdiscrimination, hamstrung by a phenomenon known as “selective adherencebias”: Recent studies suggest that the ‘human in the loop’ may be predisposedto agree with those results of the automated processing that are more alignedwith their personal pre-existing biases. Such biases may be based on sociallyinduced stereotypes, beliefs and social identities, and result in selective adoption ofalgorithmic advice. It is known that humans tend to be susceptible to confirmationbias, meaning that they assign greater weight to information congruent with prior- 3 -beliefs and less to content that contradicts them. Whereas automation bias maybe more easily detected, it may be more difficult to detect and prevent selectiveadherence bias, especially in cases where the competent authorities share thesame (e.g. regional) biases as the PIU. This holds true especially when PIUs andcompetent security authorities receive an excess of potentially false matches. Inpractice, this could result in high risks of false suspicion for members of negativelystereotyped minority groups.A decision to be rememberedThese gripes notwithstanding, the ruling does provide important guidelines onassessing other security-related instruments. Some of the aforementioned standardsmay be tailored to the PNR context. However, they pertain to features and risks thatthe PNR system shares with other security instruments designed at preventivelydetecting threats in large datapools through automated processing. Ligue des droitshumains is a decision to be remembered.In Part II of this contribution (coming 12 May 2023) we assess what it entails for theETIAS Regulation and the EU Commission’s CSAM proposal.- 4 -

Automated predictive threat detection after Ligue des Droits Humains

https://intr2dok.vifa-recht.de/servlets/MCRFileNodeServlet/mir_derivate_00015236/Automated_predictive_threat_detection_after_Ligue_des_Droits_Humains.pdf

Automated predictive threat detection after Ligue des Droits Humains

Abstract

Similar works

Full text

Available Versions

<intR>²Dok