Towards speech quality assessment using a crowdsourcing approach: evaluation of standardized methods

Subjective speech quality assessment has traditionally been carried out in laboratory environments under controlled conditions. With the advent of crowdsourcing platforms tasks, which need human intelligence, can be resolved by crowd workers over the Internet. Crowdsourcing also offers a new paradigm for speech quality assessment, promising higher ecological validity of the quality judgments at the expense of potentially lower reliability. This paper compares laboratory-based and crowdsourcing-based speech quality assessments in terms of comparability of results and efficiency. For this purpose, three pairs of listening-only tests have been carried out using three different crowdsourcing platforms and following the ITU-T Recommendation P.808. In each test, listeners judge the overall quality of the speech sample following the Absolute Category Rating procedure. We compare the results of the crowdsourcing approach with the results of standard laboratory tests performed according to the ITU-T Recommendation P.800. Results show that in most cases, both paradigms lead to comparable results. Notable differences are discussed with respect to their sources, and conclusions are drawn that establish practical guidelines for crowdsourcing-based speech quality assessment

Impact of the Number of Votes on the Reliability and Validity of Subjective Speech Quality Assessment in the Crowdsourcing Approach

The subjective quality of transmitted speech is traditionally assessed in a controlled laboratory environment according to ITU-T Rec. P.800. In turn, with crowdsourcing, crowdworkers participate in a subjective online experiment using their own listening device, and in their own working environment. Despite such less controllable conditions, the increased use of crowdsourcing micro-task platforms for quality assessment tasks has pushed a high demand for standardized methods, resulting in ITU-T Rec. P.808. This work investigates the impact of the number of judgments on the reliability and the validity of quality ratings collected through crowdsourcing-based speech quality assessments, as an input to ITU-T Rec. P.808 . Three crowdsourcing experiments on different platforms were conducted to evaluate the overall quality of three different speech datasets, using the Absolute Category Rating procedure. For each dataset, the Mean Opinion Scores (MOS) are calculated using differing numbers of crowdsourcing judgements. Then the results are compared to MOS values collected in a standard laboratory experiment, to assess the validity of crowdsourcing approach as a function of number of votes. In addition, the reliability of the average scores is analyzed by checking inter-rater reliability, gain in certainty, and the confidence of the MOS. The results provide a suggestion on the required number of votes per condition, and allow to model its impact on validity and reliability.Comment: This paper has been accepted for publication in the 2020 Twelfth International Conference on Quality of Multimedia Experience (QoMEX

An Open source Implementation of ITU-T Recommendation P.808 with Validation

The ITU-T Recommendation P.808 provides a crowdsourcing approach for conducting a subjective assessment of speech quality using the Absolute Category Rating (ACR) method. We provide an open-source implementation of the ITU-T Rec. P.808 that runs on the Amazon Mechanical Turk platform. We extended our implementation to include Degradation Category Ratings (DCR) and Comparison Category Ratings (CCR) test methods. We also significantly speed up the test process by integrating the participant qualification step into the main rating task compared to a two-stage qualification and rating solution. We provide program scripts for creating and executing the subjective test, and data cleansing and analyzing the answers to avoid operational errors. To validate the implementation, we compare the Mean Opinion Scores (MOS) collected through our implementation with MOS values from a standard laboratory experiment conducted based on the ITU-T Rec. P.800. We also evaluate the reproducibility of the result of the subjective speech quality assessment through crowdsourcing using our implementation. Finally, we quantify the impact of parts of the system designed to improve the reliability: environmental tests, gold and trapping questions, rating patterns, and a headset usage test

Human-Machine Communication: Complete Volume. Volume 6

his is the complete volume of HMC Volume 6

Technological Innovation, Data Analytics, and Environmental Enforcement

Technical innovation is ubiquitous in contemporary society and contributes to its extraordinarily dynamic character. Sometimes these innovations have significant effects on the state of the environment or on human health and they have stimulated efforts to develop second order technologies to ameliorate those effects. The development of the automobile and its impact on life in the United States and throughout the world is an example. The story of modern environmental regulation more generally includes chapters filled with examples of similar efforts to respond to an enormous array of technological advances. This Article uses a different lens to consider the role of technological innovation. In particular, it considers how technological advances have the potential to shape governance efforts in the compliance realm. The Article demonstrates that such technological advances – especially new and improved monitoring capacity, advances in information dissemination through e-reporting and other techniques, and improved capacity to analyze information – have significant potential to transform governance efforts to promote compliance. Such transformation is likely to affect not only the “how” of compliance promotion, but also the “who.” Technological innovation is likely to contribute to new thinking about the roles key actors can and should play in promoting compliance with legal norms. The Article discusses some of the potential benefits of these types of technological innovation in the context of the Environmental Protection Agency (EPA)’s ongoing efforts to improve its compliance efforts by taking advantage of emerging technologies. We also identify some of the pitfalls or challenges that agencies such as EPA need to be aware of in opening this emerging bundle of new tools and making use of them to address real-world environmental needs

Technological Innovation, Data Analytics, and Environmental Enforcement

Technical innovation is ubiquitous in contemporary society and contributes to its extraordinarily dynamic character. Sometimes these innovations have significant effects on the state of the environment or on human health and they have stimulated efforts to develop second order technologies to ameliorate those effects. The development of the automobile and its impact on life in the United States and throughout the world is an example. The story of modern environmental regulation more generally includes chapters filled with examples of similar efforts to respond to an enormous array of technological advances. This Article uses a different lens to consider the role of technological innovation. In particular, it considers how technological advances have the potential to shape governance efforts in the compliance realm. The Article demonstrates that such technological advances – especially new and improved monitoring capacity, advances in information dissemination through e-reporting and other techniques, and improved capacity to analyze information – have significant potential to transform governance efforts to promote compliance. Such transformation is likely to affect not only the “how” of compliance promotion, but also the “who.” Technological innovation is likely to contribute to new thinking about the roles key actors can and should play in promoting compliance with legal norms. The Article discusses some of the potential benefits of these types of technological innovation in the context of the Environmental Protection Agency (EPA)’s ongoing efforts to improve its compliance efforts by taking advantage of emerging technologies. We also identify some of the pitfalls or challenges that agencies such as EPA need to be aware of in opening this emerging bundle of new tools and making use of them to address real-world environmental needs

Democracy and the Media: The Year in C-SPAN Archives Research, Volume 7

Volume 7 of The Year in C-SPAN Archives Research series focuses on the relationship between democracy and the media. Using the extensive collection of the C-SPAN Video Library, chapters cover Trump political rallies, congressional references of late-night comedy, responses of African American congresswomen to COVID-19 bills, and congressional attacks on the media through floor speeches in the House of Representatives and Senate. The C-SPAN Video Library is unique because there is no other research collection that is based on video research of contemporary politics. Methodologically distinctive, much of the research uses new techniques to analyze video, text, and spoken words of political leaders. No other book examines such a wide range of topics―from immigration to climate change to race relations―using video as the basis for research