Mügge, Daniel

English

<intR>²Dok

Squaring the triangle of fundamentalrights concernsDaniel Mügge 2023-05-12T14:32:39Ex ante, the July 2022 ruling by the Court of Justice of the EU (CJEU) on PassengerName Records (PNR) had a very specific scope — the use of passenger namerecords by government agencies. Upon closer inspection, however, it hasimportant implications for the governance of algorithms more generally. That istrue especially for the proposed AI Act, which is currently working its way throughthe EU institutions. It highlights, ultimately, how national, or in this case European,legal orders may limit the scope for international regulatory harmonization andcooperation.A potential clash with jurisprudential limits on AIpoliciesFirst of all, the PNR ruling, and the changes to the PNR Directive it implies, are asimple reminder that EU legislation is open to challenge in court. Obvious as it mayseem, this consideration has hardly figured in debates about the AI Act. A widerange of stakeholders — including NGOs but also EU bodies such as the EuropeanData Protection Supervisor—have voiced concerns about AI regulation plans.Those concerns frequently revolved around potential violations of ethical norms,for example, the right to privacy or non-discrimination. Most of these standpointscombined ethical arguments about what is desirable or not with technologicalarguments about the actual effects the application of certain algorithms would have.In contrast, few arguments considered whether certain AI use cases would evenwithstand legal scrutiny by the CJEU because they might violate fundamental rights,as outlined below.This silence is remarkable. Many experts genuinely puzzle over when and wherealgorithms and fundamental rights may clash. And not only are there no easyanswers: because of the speed with which AI technologies evolve, it is easilyconceivable that legally contentious use cases emerge for which present-day law,and also a future AI Act, had not provided. (The haste with which provisions aboutgenerative AI were inserted into the negotiations at the 11th hour is instructive.)Sooner or later, legal challenges to the AI Act are to be expected, and it is anybody’sguess how those will look, and how they will be decided.The PNR ruling points to additional complications: one common discussiontopic in AI debates is the explainability of algorithmic output, especially whenalgorithms are used in public policy decisions affecting individuals. Worries aboutdiscrimination also feature widely. At least with respect to the use of PNR in lawenforcement and security contexts, however, the CJEU puts the bar even higher:- 1 -the PNR Directive itself requires that criteria for identifying subjects have to be “pre-determined” (§6.2(b)), and the Court finds that to be incompatible with self-learningalgorithms as long as the output is not transparent to humans. It thus turns the PNRDirective’s own wording against the use of algorithmic tools.At the same time, in paragraph 195 of its ruling, it cites Article 47 of the EU Charterof Fundamental Rights — the right to an effective remedy — as potentially atodds with opaque and implicit selection criteria. This interpretation suggests that“unexplainable AI” might face much broader limitations in its applicability than onlyemanating from, in this case, the PNR Directive itself. After all, algorithms’ addedvalue is to identify patterns in the data that humans would miss — self-learningalgorithms are used precisely where criterium pre-definition fails. That may spellbroader trouble for their use in public policy when potential fundamental rights are onthe line.Moreover, the ruling underlined the importance of proportionality: may potentialrights-infringements be justified in light of the security risks that they tackle? Again,this question is thorny, because the potential uses of AI vary highly in the level ofrisks that they claim to address. Blanket rules for or against employing potentiallyrights-infringing AI in law enforcement seem difficult from that angle.Irrespective of where one stands on these issues, the key is that there is significantscope for legal challenges of provisions in the AI Act. As the Schrems cases as wellas the PNR ruling have shown, these kinds of challenges may be successful andcan have momentous consequences. Up to now, there seems to be little realizationthat any compromises coming out of the AI Act trilogues might, at least in part, notwithstand legal scrutiny, either now or in the future.Extraterritorial implicationsWhile a future AI regime for Europe is being negotiated, EU representatives havealso been heavily involved in international talks and exchanges to craft AI rules.Those concern especially the OECD, whose official AI definition from 2019 recentlyemerged as a proposed compromise for the EU’s own legislation, and the Trade andTechnology Council, the forum for transatlantic policy exchange and negotiation,which largely concentrates on digital technologies. Alignment of EU policy withinternational rules, and their embedding in a sort of transatlantic regulatory regime,are both important European policy goals. What does the PNR ruling imply for suchefforts?To begin, one reading of the CJEU’s decision is that, at least for certain use cases,EU citizens’ right to effective judicial remedies as spelled out in Article 47 of theEU Charter of Fundamental Rights is incompatible with algorithms whose resultscannot be translated into clear criteria and are not certified discrimination-free. Ifso, that would generate hard limits on the AI-powered services and applicationswhich non-EU companies could offer in the EU, either directly or indirectly. TheAI Act may impose such limits on its own, as well. As an implication of the PNRruling, or its juridical spirit, these limits would swing free from the will of legislators.- 2 -In other words, the fundamental rights of EU citizens, as interpreted by the CJEU,may define the outer boundaries of regulatory cooperation in the AI field — no matterhow much goodwill there might be to find a compromise with, for example, the USA.Irrespective of whether these limits would be heeded in transatlantic or multilateralnegotiations ex ante, or would emerge later on through successful legal challenges,as happened in the Schrems cases, they might cause serious frustrations among theEU’s international partners.This logic also casts its shadow on the outsourcing of rule-making to technocraticexpert bodies, such as the European Committee for Standardization andthe European Electrotechnical Committee for Standardization (see Veale/Zuiderveen Borgesius 2021), and potentially the International Organization forStandardization (ISO) and the International Electrotechnical Commission (IEC) astheir global counterparts. Already, the relevant global standard setting committees(subcommittee 42 of the ISO/IEC) ponder to what degree they would or should enterpolitical or normative territory with their efforts, for example, to devise proceduresto determine algorithmic bias. On the one hand, the CJEU’s own interpretationof what would constitute a legally robust definition of such thorny concepts mightcompromise the formal independence of technical standard setters, given thatfundamental rights would take precedence over any “technical” compromise thelatter might devise. On the other hand, if such compromises were found to withstandlegal scrutiny, they might offer an escape from otherwise fraught legal and ethicaldebates. Either way, the scope for outsourcing standard definitions that touch onfundamental rights questions would itself depend on the view of, and potential reviewby, the CJEU.Finally, many aspects of AI technologies and their applications are bound to evolvesignificantly in the future. That includes the kind of data available to train them,technological ways to extract comprehensible “criteria” for (suggested) decisionsfrom algorithms, and forms of applying them. As is the case generally, the legalframework for AI will therefore have to be dynamic in order to accommodate futuretechnological and societal developments. This, too, limits the degree to whichlegislators or negotiators could lock the EU into particular bilateral or multilateralagreements on AI governance.Future scenariosIf the PNR ruling implies that fundamental rights of EU citizens may demarcatethe outer limits of EU-external regulatory cooperation, which scenarios does thatsuggest for the future? One scenario, somewhat surprisingly, is a form of inadvertentBrussels effect — but very different from Bradford’s original logic (Bradford 2020).Here, other parties to regulatory negotiations might appreciate, however grudgingly,that certain safeguards in EU law may be unavoidable, no matter what they thinkof them — once more, the Schrems cases are instructive. The EU’s limited roomfor manoeuvre on some of these questions may, in fact, strengthen its bargainingposition.- 3 -It is equally plausible, however, that divergent regulatory preferences andunwillingness, or inability, to compromise might generate disparate levels ofregulatory stringency even among, for example, the USA and the EU. If so,companies might “level up” to presumably higher EU standards voluntarily in theproducts they offer — a dynamic that David Vogel (1995) has dubbed the Californiaeffect, in which American car producers voluntarily embraced stringent Californianenvironmental rules across their product palette. Alternatively, markets for AI-powered products might fragment to some degree, with more or less differentversions of products on offer in different jurisdictions, each compliant with local laws.And, depending on how difficult it is to custom-tailor products to diverse regulatoryregimes, some companies might opt to forego EU market access altogether, even ifits overall market size is likely to mitigate against that approach.In the meantime, the geopolitical climate has continued to deteriorate, not only inlight of the Russian war against Ukraine, but also through souring Sino-Americanrelations. AI governance, not least in the EU itself, had initially largely been framedas a technological, commercial and societal issue. Certainly, since the publication ofthe report by the American National Security Commission on Artificial Intelligence,however, AI technologies are increasingly viewed through a security and militarylens. The EU AI Act itself steers clear of the intersection between AI and nationalsecurity, not least in light of the EU’s limited competences there. To the degree thatmore and more aspects of AI governance were to be framed as security-relevant —for example, because of AI technologies’ dual use character — the scope of the AIAct provisions and the protections they provide might shrink. It will be interesting tosee whether the CJEU, and its interpretation of fundamental rights, will then fill thatgap and provide guardrails for AI development and application that at present arehardly considered.The dilemma of fundamental rights, the need forlegal flexibility, and international agreementsTaken together, algorithms may constitute a fundamental rights governancechallenge, squeezing from three sides: first, fundamental rights as interpreted bythe CJEU impose hard limits on what is and is not permissible. Second, at the sametime, the speedy development of the technologies themselves would seem to callfor a much more open-ended and flexible legal framework. And third, geopoliticalas well as economic imperatives would seem to require the ability to commit tointernational agreements, irrespective of the former two considerations. The PNRruling suggests how difficult that triangle will be to square, not only for passengerdata, but for algorithms more generally.- 4 -

Squaring the triangle of fundamental rights concerns

https://intr2dok.vifa-recht.de/servlets/MCRFileNodeServlet/mir_derivate_00015260/Squaring_the_triangle_of_fundamental_rights_concerns.pdf

Squaring the triangle of fundamental rights concerns

Abstract

Similar works

Full text

Available Versions

<intR>²Dok