A Discussion on Building Practical NLP Leaderboards: The Case of Machine Translation

Recent advances in AI and ML applications have benefited from rapid progress in NLP research. Leaderboards have emerged as a popular mechanism to track and accelerate progress in NLP through competitive model development. While this has increased interest and participation, the over-reliance on single, and accuracy-based metrics have shifted focus from other important metrics that might be equally pertinent to consider in real-world contexts. In this paper, we offer a preliminary discussion of the risks associated with focusing exclusively on accuracy metrics and draw on recent discussions to highlight prescriptive suggestions on how to develop more practical and effective leaderboards that can better reflect the real-world utility of models.Comment: pre-print: comments and suggestions welcom

Intent Identification and Entity Extraction for Healthcare Queries in Indic Languages

Scarcity of data and technological limitations for resource-poor languages in developing countries like India poses a threat to the development of sophisticated NLU systems for healthcare. To assess the current status of various state-of-the-art language models in healthcare, this paper studies the problem by initially proposing two different Healthcare datasets, Indian Healthcare Query Intent-WebMD and 1mg (IHQID-WebMD and IHQID-1mg) and one real world Indian hospital query data in English and multiple Indic languages (Hindi, Bengali, Tamil, Telugu, Marathi and Gujarati) which are annotated with the query intents as well as entities. Our aim is to detect query intents and extract corresponding entities. We perform extensive experiments on a set of models in various realistic settings and explore two scenarios based on the access to English data only (less costly) and access to target language data (more expensive). We analyze context specific practical relevancy through empirical analysis. The results, expressed in terms of overall F1 score show that our approach is practically useful to identify intents and entities

COVID-19 and the secret virtual assistants: the social weapons for a state of emergency

Technologies are ubiquitous in modern Britain, gradually infiltrating many areas of our working and personal lives. But what role can technology play in the current COVID-19 pandemic? At a time when our usual face to face social interactions are temporarily suspended, many of us have reached out to technology (e.g. Skype, WhatsApp, Facebook, Zoom) to help maintain a sense of closeness and connection to friends, family and vital services. One largely unsung technology is the virtual assistant (VA), a cost-efficient technology enabling users to access the Internet of Things using little more than voice. Deploying an ecological framework, in the context of smart age-friendly cities, this paper explores how VA technology can function as an emergency response system, providing citizens with systems to connect with friends, family, vital services and offering assistance in the diagnosis of COVID-19. We provide an illustration of the potentials and challenges VAs present, concluding stricter regulation and controls should be implemented before VAs can be safely integrated into smart age-friendly cities across the globe

Disrupting Digital Monolingualism: A report on multilingualism in digital theory and practice

This report is about the Disrupting Digital Monolingualism virtual workshop in June 2020. The DDM workshop sought to draw together a wide range of stakeholders active in confronting the current language bias in most of the digital platforms, tools, algorithms, methods, and datasets which we use in our study or practice, and to reverse the powerful impact this bias has on geocultural knowledge dynamics in the wider world. The workshop aimed to describe the state of the art across different academic disciplines and professional fields, and foster collaboration across diverse perspectives around four points of focus: Linguistic and geocultural diversity in digital knowledge infrastructures; Working with multilingual methods and data; Transcultural and translingual approaches to digital study; and Artificial intelligence, machine learning and NLP in language worlds. Event website https://languageacts.org/digital-mediations/event/disrupting-digital-monolingualism/ This report forms part of a series of reports produced by the Digital Mediations strand of the Language Acts & Worldmaking project, in this case in collaboration with the translingual strand of the Cross-Language Dynamics project (based at the Institute of Modern Languages Research), both funded by the UK Arts and Humanities Research Council’s Open World Research Initiative. Digital Mediations explores interactions and tensions between digital culture, multilingualism and language fields including the Modern Languages

The Digital Leisure Divide and the Forcibly Displaced

UNHCR has been pursuing an agenda of enhanced connectivity and digital inclusion for forcibly displaced people. In 2020, following an array of standalone efforts in pursuing these agendas – for example, through the 2016 Connectivity for Refugees Strategy – the organization began a journey to consolidate initiatives around digital transformation into a new organization-wide strategy. One priority outcome area is around digital inclusion that seeks to ensure forcibly displaced and stateless people “have equitable access to digital technology and channels and can use them to pursue opportunities for lifelong learning, inclusion in the digital economy, leisure, and solutions.” For a number of years, many digital inclusion interventions have been tied to specific developmental goals – enhanced education, use of digital financial services, greater access to information, among others. There is emerging evidence that challenges the notion that those targeted with such interventions prioritize connectivity for these purposes. Rather, the agenda highlights leisure as a key driver for adoption of digital technologies, and a critical use case for such technologies that bring indirect benefits beyond the ‘virtuous’ aims of humanitarian aid and development programmes globally. In this report, UNHCR and Erasmus University Rotterdam (EUR) scholars document the evidence on digital leisure in the forced displacement context, highlighting issues unique to that context. This report constitutes a continuation of the desk review,1 and provides evidence from fieldwork carried out in two refugee shelters in the city of Boa Vista, Brazil – Rondon III and September 13 – at the end of 2021. The report focuses on the main uses and potential benefits of digital leisure in refugee contexts. It brings together evidence from Venezuelan forcibly displaced people with an emphasis on Brazil due to that country’s relevance in the human mobility context within the Latin American region. The report aims to inform actors in the government, private, non-profit, and aid agency sectors who are interested in digital inclusion and rights-based solutions for forcibly displaced people. It provides insights about issues of access, privacy, and trust experienced by forcibly displaced persons while using devices and navigating connectivity in their everyday lives. It also explores the opportunities for community-building and local citizenship through content creation and connection with family, friends, and society at large. We reveal how digital leisure fosters unique opportunities for self-realization and shapes specific worldviews through their information practices in digital spaces. The possible livelihoods enabled by digital leisure and the aspirational digital lives of participating Venezuelan refugees and migrants are also explored