Survey of Web-based Crowdsourcing Frameworks for Subjective Quality Assessment

The popularity of the crowdsourcing for performing various tasks online increased significantly in the past few years. The low cost and flexibility of crowdsourcing, in particular, attracted researchers in the field of subjective multimedia evaluations and Quality of Experience (QoE). Since online assessment of multimedia content is challenging, several dedicated frameworks were created to aid in the designing of the tests, including the support of the testing methodologies like ACR, DCR, and PC, setting up the tasks, training sessions, screening of the subjects, and storage of the resulted data. In this paper, we focus on the web-based frameworks for multimedia quality assessments that support commonly used crowdsourcing platforms such as Amazon Mechanical Turk and Microworkers. We provide a detailed overview of the crowdsourcing frameworks and evaluate them to aid researchers in the field of QoE assessment in the selection of frameworks and crowdsourcing platforms that are adequate for their experiments

Best Practices and Recommendations for Crowdsourced QoE - Lessons learned from the Qualinet Task Force Crowdsourcing

Crowdsourcing is a popular approach that outsources tasks via the Internet to a large number of users. Commercial crowdsourcing platforms provide a global pool of users employed for performing short and simple online tasks. For quality assessment of multimedia services and applications, crowdsourcing enables new possibilities by moving the subjective test into the crowd resulting in larger diversity of the test subjects, faster turnover of test campaigns, and reduced costs due to low reimbursement costs of the participants. Further, crowdsourcing allows easily addressing additional features like real-life environments. This white paper summarizes the recommendations and best practices for crowdsourced quality assessment of multimedia applications from the Qualinet Task Force on “Crowdsourcing”. The European Network on Quality of Experience in Multimedia Systems and Services Qualinet (COST Action IC 1003, see www.qualinet.eu) established this task force in 2012 which has more than 30 members. The recommendation paper resulted from the experience in designing, implementing, and conducting crowdsourcing experiments as well as the analysis of the crowdsourced user ratings and context data

COMPARISON OF LOSSLESS VIDEO CODECS FOR CROWD-BASED QUALITY ASSESSMENT ON TABLETS

ABSTRACT Video quality evaluation with subjective testing is both time consuming and expensive. A promising new approach to traditional testing is the so-called crowdsourcing, moving the testing effort into the Internet. The advantages of this approach are not only the access to a larger and more diverse pool of test subjects, but also the significant reduction of the financial burden. Extending this approach to tablets, allows us not only to assess video quality in a realistic environment for an ever more important use case, but also provides us with a well-defined hardware platform, eliminating on of the main drawbacks of crowdsourced video quality assessment. One prerequisite, however, is the support of lossless coding on the used tablets. We therefore examine in this contribution the performance of lossless video codecs on the iPad platform. Our results show, that crowdbased video testing is already feasible for CIF-sized videos on tablets, but also that there may be limits for higher resolution videos

Drug release from thin films encapsulated by a temperature-responsive hydrogel

Control over drug delivery may be interestingly achieved by using temperature responsive encapsulants, which change their thickness and mesh size with temperature. The prototype N-isopropylacrylamide hydrogel cross-linked with di(ethylene glycol) divinyl ether p(NIPAAm-co-DEGDVE) swells at low temperature and collapses above the lower critical solution temperature (LCST), ∼29 °C in a buffer. It might be expected that drug release from such encapsulation is always favored below the LCST, due to the larger free volume present in the swollen polymer film. Recent results show contradicting behavior where some cases behave as expected and others release much less when the polymer layer is swollen. In this study, layers of the drugs phenytoin, clotrimazole and indomethacin were drop cast on glass and p(NIPAAM-co-DEGDVE) layers were then synthesized directly on top of these drug layers via initiated chemical vapor deposition (iCVD), a solvent-free and gentle polymerization technique. Dissolution experiments were then performed, in which the drug release through the hindrance of the hydrogel was measured at different pH values. The results show that not only the swelling but also the permeate (drug in this case)-polymer interaction plays an important role in the release

Crowdsourcing-based multimedia subjective evaluations: a case study on image recognizability and aesthetic appeal

International audienceResearch on Quality of Experience (QoE) heavily relies on subjective evaluations of media. An important aspect of QoE concerns modeling and quantifying the subjective notions of 'beauty' (aesthetic appeal) and 'something well-known' (content recognizability), which are both subject to cultural and social effects. Crowdsourcing, which allows employing people worldwide to perform short and simple tasks via online platforms, can be a great tool for performing subjective studies in a time and cost-effective way. On the other hand, the crowdsourcing environment does not allow for the degree of experimental control which is necessary to guarantee reliable subjective data. To validate the use of crowdsourcing for QoE assessments, in this paper, we evaluate aesthetic appeal and recognizability of images using the Microworkers crowdsourcing platform and compare the outcomes with more conventional evaluations conducted in a controlled lab environment. We find high correlation between crowdsourcing and lab scores for recognizability but not for aesthetic appeal, indicating that crowdsourcing can be used for QoE subjective assessments as long as the workers' tasks are designed with extreme care to avoid misinterpretations

The DLR Transport and the Environment Project - Building competency for a sustainable mobility future

This article describes the thematic and organizational approach of the DLR Transport Program’s research project “Transport and the Environment” (Verkehrsentwicklung und Umwelt, VEU). It illuminates the research approach, which employs scenario techniques to achieve a common understanding and a framework for the project. Establishing a platform for scientific exchange, VEU clearly facilitates the interdisciplinary integration of research on mobility within the DLR and its partner institutes. A set of transport scenarios for Germany and Europe (time horizon 2040), which are presented in greater detail in this article, is used in this respect and aims to provide answers for pressing societal questions