Sound collection systems using a crowdsourcing approach to construct sound map based on subjective evaluation

This paper presents a sound collection system that uses crowdsourcing to gather information for visualizing area characteristics. First, we developed a sound collection system to simultaneously collect physical sounds, their statistics, and subjective evaluations. We then conducted a sound collection experiment using the developed system on 14 participants. We collected 693,582 samples of equivalent Aweighted loudness levels and their locations, and 5,935 samples of sounds and their locations. The data also include subjective evaluations by the participants. In addition, we analyzed the changes in sound properties of some areas before and after the opening of a large-scale shopping mall in a city. Next, we implemented visualizations on the server system to attract users’ interests. Finally, we published the system, which can receive sounds from any Android smartphone user. The sound data were continuously collected and achieved a specified result

When is it Better to Compare than to Score?

When eliciting judgements from humans for an unknown quantity, one often has the choice of making direct-scoring (cardinal) or comparative (ordinal) measurements. In this paper we study the relative merits of either choice, providing empirical and theoretical guidelines for the selection of a measurement scheme. We provide empirical evidence based on experiments on Amazon Mechanical Turk that in a variety of tasks, (pairwise-comparative) ordinal measurements have lower per sample noise and are typically faster to elicit than cardinal ones. Ordinal measurements however typically provide less information. We then consider the popular Thurstone and Bradley-Terry-Luce (BTL) models for ordinal measurements and characterize the minimax error rates for estimating the unknown quantity. We compare these minimax error rates to those under cardinal measurement models and quantify for what noise levels ordinal measurements are better. Finally, we revisit the data collected from our experiments and show that fitting these models confirms this prediction: for tasks where the noise in ordinal measurements is sufficiently low, the ordinal approach results in smaller errors in the estimation

Towards Participatory Design of City Soundscapes

Publisher Copyright: © 2021, The Author(s), under exclusive license to Springer Nature Switzerland AG.Sonic environments of fast-growing urban areas are an integral part of the quality of our everyday living in cities. Due to the individual nature of the sonic experience, collecting and analyzing such experiences needs methods for gathering accurate and useful data about them. This paper describes how to incorporate the concept of soundscape into city planning processes. To achieve this, we propose creating participatory methods for gathering data from the citizens so that the data would be useful and relevant for the city planning professionals. Since the participatory planning process aims at in evolving the citizens, we suggest methods that utilize crowdsourcing, mobile technology and machine learning for presenting, workshopping, and designing soundscapes in the city context.Peer reviewe

Coco-Nut: Corpus of Japanese Utterance and Voice Characteristics Description for Prompt-based Control

In text-to-speech, controlling voice characteristics is important in achieving various-purpose speech synthesis. Considering the success of text-conditioned generation, such as text-to-image, free-form text instruction should be useful for intuitive and complicated control of voice characteristics. A sufficiently large corpus of high-quality and diverse voice samples with corresponding free-form descriptions can advance such control research. However, neither an open corpus nor a scalable method is currently available. To this end, we develop Coco-Nut, a new corpus including diverse Japanese utterances, along with text transcriptions and free-form voice characteristics descriptions. Our methodology to construct this corpus consists of 1) automatic collection of voice-related audio data from the Internet, 2) quality assurance, and 3) manual annotation using crowdsourcing. Additionally, we benchmark our corpus on the prompt embedding model trained by contrastive speech-text learning.Comment: Submitted to ASRU202

Emerging Opportunities: Monitoring and Evaluation in a Tech-Enabled World

Various trends are impacting on the field of monitoring and evaluation in the area of international development. Resources have become ever more scarce while expectations for what development assistance should achieve are growing. The search for more efficient systems to measure impact is on. Country governments are also working to improve their own capacities for evaluation, and demand is rising from national and community-based organizations for meaningful participation in the evaluation process as well as for greater voice and more accountability from both aid and development agencies and government.These factors, in addition to greater competition for limited resources in the area of international development, are pushing donors, program participants and evaluators themselves to seek more rigorous – and at the same time flexible – systems to monitor and evaluate development and humanitarian interventions.However, many current approaches to M&E are unable to address the changing structure of development assistance and the increasingly complex environment in which it operates. Operational challenges (for example, limited time, insufficient resources and poor data quality) as well as methodological challenges that impact on the quality and timeliness of evaluation exercises have yet to be fully overcome

An Exploration of the Application of Crowdsourcing to Health-Related Research

Background: A growing number of health research projects are employing crowdsourcing as part of their methods, leveraging it to inform everything from study design to participant recruitment to data collection and analysis. Therefore, greater understanding of how crowdsourcing is being used and how it can be applied in the research contexts warrants further exploration. Purpose: The purpose of this dissertation was to explore crowdsourcing as a means of research inquiry, and to locate it amidst research paradigms; understand how crowdsourcing in research is used in practice; and, create a framework, and guidelines, for researchers using crowdsourcing in their research. Research Questions: The following research questions were posed: a) What are the core principles and philosophies of crowdsourcing as a research paradigm? b) How and why are researchers using crowdsourcing? c) How are researchers addressing the basic characteristic of crowdsourcing in research studies? d) How could researcher address the basic characteristics of crowdsourcing in research studies? Methodology: To answer the first question, the ontology, epistemology, methodology and axiology of crowdsourcing as a research paradigm was explored. An observational study then analyzed 227 publically available research projects on a crowdsourcing website. Finally, a modified Delphi technique was used to determine whether there was a consensus among 18 experts regarding the use of crowdsourcing for the purposes of research. Based on these studies, a conceptual framework for crowdsourcing research studies emerged. Findings: The core principles and philosophies of crowdsourcing resemble those of the participatory paradigm. Crowdsourcing is being used primarily as a method for participant recruitment, data collection and analysis. The most plausible framework for the application of crowdsourcing in studies is based on the research paradigm which in turn defines the roles of the crowd. The role of the crowd defined in generally acceptable research terms (i.e. participant, data collection, analysis, study design etc.) makes it feasible to align the role with the research paradigms to define the crowd as subjects or participants, citizen scientists, or co-researchers. Implications: These findings suggest that crowdsourcing as a method should align with the research paradigm within which it is being applied. Implications for future research are discussed

Leveraging Mixed Expertise in Crowdsourcing.

Crowdsourcing systems promise to leverage the "wisdom of crowds" to help solve many kinds of problems that are difficult to solve using only computers. Although a crowd of people inherently represents a diversity of skill levels, knowledge, and opinions, crowdsourcing system designers typically view this diversity as noise and effectively cancel it out by aggregating responses. However, we believe that by embracing crowd workers' diverse expertise levels, system designers can better leverage that knowledge to increase the wisdom of crowds. In this thesis, we propose solutions to a limitation of current crowdsourcing approaches: not accounting for a range of expertise levels in the crowd. The current body of work in crowdsourcing does not systematically examine this, suggesting that researchers may not believe the benefits of using mixed expertise warrants the complexities of supporting it. This thesis presents two systems, Escalier and Kurator, to show that leveraging mixed expertise is a worthwhile endeavor because it materially benefits system performance, at scale, for various types of problems. We also demonstrate an effective technique, called expertise layering, to incorporate mixed expertise into crowdsourcing systems. Finally, we show that leveraging mixed expertise enables researchers to use crowdsourcing to address new types of problems.PhDComputer Science and EngineeringUniversity of Michigan, Horace H. Rackham School of Graduate Studieshttp://deepblue.lib.umich.edu/bitstream/2027.42/133307/1/afdavid_1.pd

Capturing perceived everyday lived landscapes through gamification and active crowdsourcing

Summary Landscapes are distinguishable areas of the earth with distinct characters comprised of tangible and intangible dimensions and entities. Interactions between humans and landscapes influence social, physical and mental well-being as well as guide behaviour. Understanding how landscapes are perceived has thus gained traction in sustainable and inclusive policy and decision making processes and public participation is called for. The recognised importance of understanding landscapes from an experiential and perceptual perspective and incorporating public participation in data generation efforts is reflected in overarching conventions, policy guidelines and frameworks including the European Landscape Convention (ELC), the Millennium Ecosystem Assessment (MEA), Natures Contributions to People (NCP) and the Landscape Character Assessment (LCA) framework. Major challenges for these conventions and frameworks are 1) how to collect data on landscape experiences and perceptions from a diverse group of individuals, 2) how to integrate and link physical entities, sensory experiences and intangible dimensions of landscapes and 3) how to identify other potential sources of landscape relevant information. The abundance of storage space and the accessibility of broadband internet have led to a burgeoning of user generated natural language content. In parallel, various paradigms of exploiting ubiquitous internet access for research purposes have emerged, including crowdsourcing, citizen science, volunteered geographic information and public participation geographic information systems. These low cost approaches have shown great potential in generating large amounts of data, however, they struggle with motivating and retaining participants. Gamification - broadly defined as adding entertaining or playful elements to applications or processes - has been found to increase user motivation and has explicitly been called for in landscape perception and preference research to diversify participant demographics. Meanwhile, natural language has been found to be deeply intertwined with thought and emotion and has been identified as a rich source of semantic data on how landscapes are perceived and experienced. Written texts and the ways in which these can be analysed have gained particular interest. Therefore, the overall goal of this thesis is to develop and implement a gamified crowdsourcing application to collect natural language landscape descriptions and to analyse and explore the contributions in terms of how landscapes are perceived through sensory experiences and how additional landscape relevant natural language can be identified. To approach this goal, I first elicit key data and feature requirements to collect landscape relevant information from a heterogeneous audience. Guided by the identified requirements, I develop and implement Window Expeditions, a gamified active crowdsourcing platform geared towards collecting natural language descriptions of everyday lived landscapes. The generated corpus of natural language is explored using computational methods and I present and discuss the results in light of who the contributors are, the locations from which participants contribute and salient terms found in English and German. In a further step I annotate a subset of English contributions according to the contained biophysical elements, sensory experiences and cultural ecosystem (dis)services and explore these in terms of how they are linked. Finally, I present a novel approach of using a curated high quality landscape specific dataset to computationally identify similar documents in other corpora using sentence-transformers. Using the Mechanics, Dynamics and Aesthetics (MDA) framework, the aesthetics of discovery, expression and fellowship were identified as most fitting for an active crowdsourcing platform. In addition, four groups of main dynamics were found, namely general dynamics of user interactions, contribution dynamics, exploration dynamics and moderation dynamics. The application was gamified by introducing points and leader boards and the platform was implemented in German and English (with French being added at a later point) to collect landscape descriptions in multiple languages. Demographic information was collected about the users including their year of birth, their gender, if they were at home whilst contributing and what languages users believed to be fluent in. Using the Mechanics, Dynamics and Aesthetics (MDA) framework, the aesthetics of discovery, expression and fellowship were identified as most fitting for an active crowdsourcing platform. In addition, four groups of main dynamics were found, namely general dynamics of user interactions, contribution dynamics, exploration dynamics and moderation dynamics. The application was gamified by introducing points and leader boards and the platform was implemented in German and English (with French being added at a later point) to collect landscape descriptions in multiple languages. Demographic information was collected about the users including their year of birth, their gender, if they were at home whilst contributing and what languages users believed to be fluent in reporting not being at home (n = 172) who were more likely to contribute from areas of herbaceous vegetation. Terms describing salient elements of everyday lived environments such as "tree", "house", "garden" and "street", as well as weather related phenomena and colours were found frequently in both English and German contributions in the generated corpus. Further, terms related to space, time and people were found significantly more frequently in the generated corpus compared to general natural language and representative landscape image descriptions highlighting the importance of spatial features as well as people and the times at which these were observed. Notably, descriptions referring to trees and birds were frequently found in the contributed texts, underlining their saliency in everyday lived landscapes. The results show biophyiscal terms related to vegetation (n = 556) and the built environment (n = 468) as well as weather related terms (n = 452) to be most prominent. Further, contributions referencing visual (n = 186) and auditory (n = 96) sensory experiences were found most often with positive sensory experiences being most common (n = 168) followed by neutral (n = 86) and negative (n = 68). In regards to the intangible dimensions captured in the contributed landscape descriptions, recreation (n = 68) was found most often followed by heritage (n = 36), identity (n = 26) and tranquillity (n = 23). Through linking biophysical elements, sensory experiences and cultural ecosystem (dis)services, the results show that the biophysical category of animals appears often with the sensory experience of smell/taste and the biophysical category of moving objects appears more than expected with the sensory experience of sound. Further, the results show the cultural ecosystem service of inspiration to often appear with the biophysical category of natural features and tranquillity with weather. Using a curated subcorpus of English natural language landscape descriptions (n = 428) collected with Window Expeditions, similar documents in other collections were identified. Through translating documents to vectors by means of sentence-transformers and calculating cosine similarity scores, a total of 6075 to 8172 documents were identified to be similar to contributions to Window Expeditions, depending on if the initial dataset was prefiltered for biophysical noun lemmas (a list of biophysical landscape elements derived from the Window Expeditions corpus) and Craik’s list adjectives (a list of common adjectives used to describe landscapes). Latent Dirichlet allocation topic modelling, a clustering approach which is commonly used to identify overarching topics or themes in collections of natural language, shows four distinct clusters in both Window Expeditions as well as in the corpus of identified similar documents, namely urban and residential, rural and natural, autumn and colours and snow and weather. Overall, the results presented in this thesis provide further evidence to work that natural language is a rich source of landscape specific information, capturing underlying semantics of a multitude of referenced landscape dimensions. In particular, this thesis demonstrates that computationally aided approaches to analysing and exploring landscape relevant textual data can give detailed insights into salient features of landscapes and how individuals perceive and experience these. Especially when complemented by human annotation, natural language landscape descriptions are a welcome source of data about a landscape’s biophysical elements, individual sensory experiences in landscapes and the perceived cultural ecosystem (dis)services. The findings of this thesis are accompanied by various limitations, chief amongst which are the possibilities of users to falsify their locations, the rather small amount of data that was collected through Window Expeditions and the Eurocentric definitions and approaches common in landscape perception research. The former two limitations can be addressed through implementational reiterations and promotional efforts, whereas the latter limitation calls for further consideration of the socio-culturally induced construction of landscape perception research and a rethinking of holistic approaches, especially in multicultural participatory contexts. The work presented in this thesis shows great potential in complementing landscape perception research with gamified methods of data generation. Active crowdsourcing can be a cost efficient and scalable approach of generating much needed data from a diverse audience. Exploring landscape relevant natural language with both quantitative and qualitative methods from various disciplines including geographic information science, linguistics and machine learning can lead to new insights into landscape perception, sensory landscape experiences and how these are expressed

Usability of disaster apps : understanding the perspectives of the public as end-users : a dissertation presented in partial fulfilment of the requirements for the degree of Doctor of Philosophy in Emergency Management at Massey University, Wellington, New Zealand

Listed in 2020 Dean's List of Exceptional ThesesMultiple smartphone applications (apps) exist that can enhance the public’s resilience to disasters. Despite the capabilities of these apps, they can only be effective if users find them usable. Availability does not automatically translate to usability nor does it guarantee continued usage by the target users. A disaster app will be of little or no value if a user abandons it after the initial download. It is, therefore, essential to understand the users’ perspectives on the usability of disaster apps. In the context of disaster apps, usability entails providing the elements that effectively facilitate users in retrieving critical information, and thus enabling them to make decisions during crises. Establishing good usability for effective systems relies upon focussing on the user whereby technological solutions match the user’s needs and expectations. However, most studies on the usability of disaster context technologies have been conducted with emergency responders, and only a few have investigated the publics’ perspectives as end-users. This doctoral project, written within a ‘PhD-thesis-with-publication’ format, addresses this gap by investigating the usability of disaster apps through the perspectives of the public end-users. The investigation takes an explicitly perceived usability standpoint where the experiences of the end-users are prioritised. Data analysis involved user-centric information to understand the public’s context and the mechanisms of disaster app usability. A mixed methods approach incorporates the qualitative analysis of app store data of 1,405 user reviews from 58 existing disaster apps, the quantitative analysis of 271 survey responses from actual disaster app users, and the qualitative analysis of usability inquiries with 18 members of the public. Insights gathered from this doctoral project highlight that end-users do not anticipate using disaster apps frequently, which poses particular challenges. Furthermore, despite the anticipated low frequency of use, because of the life-safety association of disasters apps, end-users have an expectation that the apps can operate with adequate usability when needed. This doctoral project provides focussed outcomes that consider such user perspectives. First, an app store analysis investigating user reviews identified new usability concerns particular to disaster apps. It highlighted users’ opinion on phone resource usage and relevance of content, among others. More importantly, it defined a new usability factor, app dependability, relating to the life-safety context of disaster apps. App dependability is the degree to which users’ perceive that an app can operate dependably during critical scenarios. Second, the quantitative results from this research have contributed towards producing a usability-continuance model, highlighting the usability factors that affect end-users’ intention to keep or uninstall a disaster app. The key influences for users’ intention to keep disaster apps are: (1) users’ perceptions as to whether the app delivers its function (app utility), (2) whether it does so dependably (app dependability), and (3) whether it presents information that can be easily understood (user-interface output). Subsequently, too much focus on (4) user-interface graphics and (5) user-interface input can encourage users to uninstall apps. Third, the results from the qualitative analysis of the inquiry data provide a basis for developing guidelines for disaster app usability. In the expectation of low level of engagement with disaster app users, the guidelines list recommendations addressing information salience, cognitive load, and trust. This doctoral project provides several contributions to the body of knowledge for usability and disaster apps. It reiterates the importance of investigating the usability of technological products for disasters and showcases the value of user-centric data in understanding usability. It has investigated usability with particular attention to the end-users’ perspectives on the context of disaster apps and, thus, produces a theoretical usability-continuance model to advance disaster app usability research and usability guidelines to encourage responsible design in practice