Towards speech quality assessment using a crowdsourcing approach: evaluation of standardized methods

Subjective speech quality assessment has traditionally been carried out in laboratory environments under controlled conditions. With the advent of crowdsourcing platforms tasks, which need human intelligence, can be resolved by crowd workers over the Internet. Crowdsourcing also offers a new paradigm for speech quality assessment, promising higher ecological validity of the quality judgments at the expense of potentially lower reliability. This paper compares laboratory-based and crowdsourcing-based speech quality assessments in terms of comparability of results and efficiency. For this purpose, three pairs of listening-only tests have been carried out using three different crowdsourcing platforms and following the ITU-T Recommendation P.808. In each test, listeners judge the overall quality of the speech sample following the Absolute Category Rating procedure. We compare the results of the crowdsourcing approach with the results of standard laboratory tests performed according to the ITU-T Recommendation P.800. Results show that in most cases, both paradigms lead to comparable results. Notable differences are discussed with respect to their sources, and conclusions are drawn that establish practical guidelines for crowdsourcing-based speech quality assessment

E³DOAS: balancing QoE and energy-saving for multi-device adaptation in future mobile wireless video delivery

Smart devices (e.g. smartphones, tablets, smart-home devices, etc.) have become important companions to most people in their daily activities, and are very much used for multimedia content exchange (i.e. video sharing, real-time/non-real-time multimedia streaming), contributing to the exponential increase in mobile traffic over the current wireless networks. While the next generation mobile networks will provide higher capacity than the current 4G systems, the network operators will face important challenges associated with the outstanding increase of both video traffic and user expectations in terms of their levels of perceived quality or Quality of Experience (QoE). Furthermore, the heterogeneity of mobile devices (e.g. screen resolution, battery life, hardware performance) also impacts severely the end-user QoE. In this context, this paper proposes an Evolved QoE-aware Energy-saving Device-Oriented Adaptive Scheme (E3DOAS ) for mobile multimedia delivery over future wireless networks. E3DOAS makes use of a coalition game-based rate allocation strategy within the multi-device heterogeneous environment, and optimizes the trade-off between the end-user perceived quality of the multimedia delivery and the mobile device energy-saving. Testing has involved a prototype of E3DOAS, a crowd-sourcing-based QoE assessment method to model non-reference perceptual video quality, and an energy measurement testbed introduced to collect power consumption parameters of the mobile devices. Simulation-based performance evaluation showed how E3DOAS outperformed other state of the art multimedia adaptive solutions in terms of energy saving, end-to-end Quality of Service (QoS) metrics and end-user perceived quality

Quality of Experience (QoE) and Quality of Service (QoS) in UAV Systems

APPEAL FROM CONVICTIONS FOR UNLAWFUL POSSESSION OF METHAMPHETAMINE, A THIRD DEGREE FELONY, IN VIOLATION OF UTAH CODE ANN. §58-37-8(2)(a)(I) (1991); UNLAWFUL POSSESSION OF MARIJUANA, A CLASS B MISDEMEANOR, IN VIOLATION OF UTAH CODE ANN. §58-37-8(2)(a)(I) (1991); AND UNLAWFUL POSSESSION OF DRUG PARAPHERNALIA, A CLASS B MISDEMEANOR, IN VIOLATION OF UTAH CODE ANN. §58-37a-5 (1981), IN THE THIRD DISTRICT COURT, DIVISION 1, OF SALT LAKE COUNTY, STATE OF UTAH, THE HONORABLE TYRONE E. MEDLEY, DISTRICT COURT JUDGE, PRESIDING

Survey of Web-based Crowdsourcing Frameworks for Subjective Quality Assessment

The popularity of the crowdsourcing for performing various tasks online increased significantly in the past few years. The low cost and flexibility of crowdsourcing, in particular, attracted researchers in the field of subjective multimedia evaluations and Quality of Experience (QoE). Since online assessment of multimedia content is challenging, several dedicated frameworks were created to aid in the designing of the tests, including the support of the testing methodologies like ACR, DCR, and PC, setting up the tasks, training sessions, screening of the subjects, and storage of the resulted data. In this paper, we focus on the web-based frameworks for multimedia quality assessments that support commonly used crowdsourcing platforms such as Amazon Mechanical Turk and Microworkers. We provide a detailed overview of the crowdsourcing frameworks and evaluate them to aid researchers in the field of QoE assessment in the selection of frameworks and crowdsourcing platforms that are adequate for their experiments