York, Jillian C.

<intR>²Dok

The moderation of extremist content isprone to error, causing real-world harmJillian C. York 2022-02-08T10:55:37In the mid-1990s, a questionable and now-debunked study from an undergraduatestudent about the prevalence of pornographic imagery online sparked a TIMEMagazine cover story, leading to a Congressional inquiry and eventually to thecreation of the Communications Decency Act (CDA). The Act included a provisionthat imposed sanctions on anyone who “uses an interactive computer serviceto send to a specific person or persons under 18 years of age or […] uses anyinteractive computer service to display in a manner available to a person under18 years of age, any comment, request, suggestion, proposal, image, or othercommunication that, in context, depicts or describes, in terms patently offensive asmeasured by contemporary community standards, sexual or excretory activities ororgans.”Then-president Bill Clinton signed the Act into law in 1996, prompting oppositionfrom civil rights groups. The American Civil Liberties Union (ACLU) filed a lawsuit,arguing that the censorship provisions of the CDA were unconstitutional becausethey would criminalize protected expression, as well as because the terms“indecency” and “patently offensive” were unconstitutionally vague. In a landmark1997 ruling, the Supreme Court ruled that the CDA created an “unacceptablyheavy burden on protected speech” that threatened to “torch a large segment ofthe Internet community.” In the decision, Justice John Paul Stevens wrote that “theinterest in encouraging freedom of expression in a democratic society outweighs anytheoretical but unproven benefit of censorship.”1)Jillian C. York, Silicon Values: TheFuture of Free Speech Under Surveillance Capitalism (Verso: 2021), p. 147.The CDA was overturned and pornography eventually proliferated across theinternet, but the debacle nevertheless had a lasting impact, creating a bright dividingline between sexually explicit websites and the rest of the web. By the time “Web2.0” came around, most major sites had in place anti-obscenity policies, even in theabsence of most other restrictions on speech.As such, while pornography remains prevalent across the web, there has been abroad chilling effect on sexual expression, as well as the expression of non-sexualnudity and sexual health information. Government and company policies intended tokeep lascivious content away from the eyes of minors (such as Facebook’s complexcommunity standard around nudity and sex) ultimately end up casting a wide net,preventing a range of expression and information-sharing.- 1 -Old tactics, new purposeToday, we are witnessing a similar phenomenon when it comes to extremist andterrorist content online: Policies intended to limit the ability of terrorist groups toorganize, recruit, and incite—as well as for individuals to praise such groups—havebeen expanded in recent years and often result in the erasure of not only extremistexpression, but human rights documentation, counterspeech, and art.While some restrictions are legally required, many corporate anti-extremism policieswere created in response to pressure from NGOs, governments, and the publicand have little or no basis in law. For instance, while groups such as the IranianRevolutionary Guard Corps (IRGC) are sanctioned by the United States andtherefore cannot be hosted by U.S. platforms, there is no law preventing individualsfrom discussing the merits of their role in Iranian society.Despite that, companies like Facebook and Google have taken a particularly hard-line approach in recent years against such groups, relying on similar tools and tacticsthat have long been used to moderate sexually explicit content.In the years following the repeal of the CDA, it was not only sexually explicit adultmaterial that proliferated across the web, but also child sexual abuse imagery(CSAM). The need to swiftly remove such content (while causing as little harm tohuman content moderators as possible) required the creation of tools that couldidentify CSAM and automate its removal. The existence of a law enforcementdatabase of CSAM made this relatively easy, as images that showed up online couldbe matched to those in an existing database. Thus, PhotoDNA — a technologythat identifies CSAM and matches it to material in a database based on uniquefingerprints, or hashes — was born. When images are detected, they are reported tothe National Center for Missing and Exploited Children (NCMEC) as required by law.The creator of PhotoDNA, Dr. Hany Farid — a professor at the University ofCalifornia, Berkeley — suggested its use for detecting terrorist imagery, initiallyto little interest. But as the use of social media by groups such as the IslamicState increased in both sophistication and scale, companies began to adopt thetechnology, relying on a database run by the Global Internet Forum to CounterTerrorism (GIFCT). Though GIFCT member companies are not required to use thedatabase, most do, at least in part.The trouble with defining ‘terrorism’The race by companies to eradicate extremism is arguably more complex than theone to fight child sexual abuse imagery, however, and for a number of reasons. First,there is no globally agreed-upon definition of “terrorism,” and throughout modernhistory, states have used the term to classify and deny rights to their opponents. Aquick glance at the United States, European, and United Nations lists of terrorismorganizations shows substantive differences in approach.- 2 -Second, most major social media platforms are subject to U.S. law in some wayor another, whether or not they are headquartered in the United States. Theymust comply with certain U.S. sanctions but are also under pressure to defaultto U.S. classifications—classifications which are often political in nature.2)AlexSchmid, “Terrorism: The Definitional Problem,” Case Western Reserve Journal ofInternational Law, 2004: Vol. 36, Issue 2, https://scholarlycommons.law.case.edu/cgi/viewcontent.cgi?article=1400&context=jil Without an internationally agreed-upondefinition of terrorism, companies—and by extension, the public—are forced to trustthe GIFCT and its member companies’ definition.This has proven troublesome. All areas of content moderation include some errors,whether the moderation is undertaken by humans, artificial intelligence, or somecombination of the two, and companies are not typically forthcoming about their errorrates. When it comes to online extremism specifically, there is reason to believe thatover-moderation is quite common, owing to the aforementioned complexities and therigidity and typically binary nature of how content policies are enforced.There are numerous illustrative examples that demonstrate the complexities ofmoderating extremist imagery. In 2017, for instance, an Emirati journalist posted aphoto which featured a picture of Hezbollah leader Hassan Nasrallah with a rainbowPride flag overlaid across it, intended as a satirical statement. Although satire is apermitted exception to the company’s rules against terrorist content, the image wasremoved because it contained the photo of a designated terrorist.3)York, SiliconValues, p. 387Documents leaked to the Guardian around the same time period demonstrated thatFacebook moderators are trained to remove imagery that contains support, praise,or representation of terrorist groups, and to ignore those that are presented in aneutral or critical fashion. But human moderators must make split-second decisions,and must therefore memorize the faces of numerous designated individuals. Thepropensity for error is apparent.Automation is error-proneAutomation seems even more prone to error than humans in this area. Training datalibraries can create normative attributes for certain types of image classifications; forinstance, a body with large breasts is assumed to belong to a woman.4)York, SiliconValues, p. 388 The image of Nasrallah, then, presented without comment, would notbe read as satirical if the machine learning algorithm was not trained to pick up onthe overlaid rainbow flag.Training a machine learning algorithm for the purpose of removing terrorist imageryrequires the creation of a dataset that includes a significant amount of content inone category, which is then fed to the algorithm for training. For example, in orderto accurately identify extremist content, a company like YouTube would create a setof data that it defines as extremist—such as a large number of ISIS videos—thenfeed that data to its algorithm. Any mistakes made by the algorithm can be difficult- 3 -to understand—unless specifically designed to be interpretable, machine learningalgorithms cannot be understood by humans.Moderating text using machine-learning algorithms can be even more complex thanmoderating imagery. The Open Technology Institute describes the challenge thus:“In the case of content such as extremist content and hate speech, there are a rangeof nuanced variations in speech related to different groups and regions, and thecontext of this content can be critical in understanding whether or not it should beremoved. As a result, developing comprehensive datasets for these categories ofcontent is challenging, and developing and operationalizing a tool that can be reliablyapplied across different groups, regions, and sub-types of speech is also extremelydifficult. In addition, the definition of what types of speech fall under these categoriesis much less clear.” The Institute concludes that “these tools are limited in that theyare unable to comprehend the nuances and contextual variations present in humanspeech.”It is not clear how much of the GIFCT database consists of human-identified versusmachine-identified content, namely because the database has not been shared withany members of civil society focused on human rights (despite demands), and theGIFCT has offered minimal information about how the database functions. Unlikethe databases used for identifying CSAM, the GIFCT database has no externaloversight.Thus, any error contained within the database is multiplied across the social web.And such errors abound: In addition to satirical content such as that in the previousexample, documentation of human rights violations and violent conflict is regularlyremoved as well. According to Syrian Archive, a group that documents and archivessuch content, at least 206,077 videos documenting rights violations were madeunavailable on YouTube between 2011 and May 2019.It is no surprise that companies have undertaken such drastic measures againstextremist content. Politicians, law enforcement, and other officials regularly engagein hyperbole that encourages strong action whilst ignoring the potential pitfalls. Anoft-repeated quote from Commander Dean Haydon, of the Met’s Counter TerrorismCommand states that “Every tweet has the potential to radicalize vulnerable people,”while terms like “eradicate” and “eliminate” have been used by efforts such as theChristchurch Call, an effort put forward by the governments of New Zealand andFrance following the attacks on a Christchurch mosque by a white supremacist in2019.The development of the Christchurch Call, a multi-stakeholder effort that includesgovernments, companies, and civil society and which works closely with the GIFCT,raises another key issue; that decisions about who is or not a terrorist, and whogets to make such a decision. The Christchurch Call was created in response to awhite supremacist act of terror, and yet the focus of GIFCT—and ostensibly of thedatabase it oversees—has been on Islamist groups such as ISIS and Al Qaeda. Thisdemonstrates a clear disconnect in goals but also illuminates a core problem with theeffort to “eradicate” terrorism: The world is not in agreement about what constitutesterrorism and, as previously mentioned, various (or perhaps most) governments- 4 -have manipulated this term to suit their political goals. As such, the lack of oversightand minimal expert and civil society involvement is exceptionally troubling.References• Jillian C. York, Silicon Values: The Future of Free Speech Under SurveillanceCapitalism (Verso: 2021), p. 147.• Alex Schmid, “Terrorism: The Definitional Problem,” Case WesternReserve Journal of International Law, 2004: Vol. 36, Issue 2, https://scholarlycommons.law.case.edu/cgi/viewcontent.cgi?article=1400&context=jil• York, Silicon Values, p. 387• York, Silicon Values, p. 388- 5 -

The moderation of extremist content is prone to error, causing real-world harm

Abstract

Similar works

Full text

Available Versions

<intR>²Dok