Estimation of quality scores from subjective tests : beyond subjects' MOS

Subjective tests for the assessment of the quality of experience (QoE) are typically run with a pool of subjects providing their opinion score using a 5-level scale. The subjects? Mean Opinion Score (MOS) is generally assumed as the best estimation of the average score in the target population. Indeed, for a large enough sample we can assume that the mean of the variations across the subjects approaches zero, but this is not the case for the limited number of subjects typically considered in subjective tests. In this paper we propose a model for the estimation of the population average QoE. We apply such model to a dataset composed of the individual scores assigned by 25 subjects to a set of gaming videos evaluated under different resolutions and compression rates. The model recognizes the ordinal multinomial nature of the data and allows for correlation between scores of the same subject on different data. The resulting estimated average QoE is shown to follow more credible patterns than the MOS, in particular with respect to improved compression rates, for which model estimates present a more coherent behaviour. In order to favour reproducibility and application for different datasets, the software that implements the model is also made publicly available

Understanding customers' holistic perception of switches in automotive human–machine interfaces

For successful new product development, it is necessary to understand the customers' holistic experience of the product beyond traditional task completion, and acceptance measures. This paper describes research in which ninety-eight UK owners of luxury saloons assessed the feel of push-switches in five luxury saloon cars both in context (in-car) and out of context (on a bench). A combination of hedonic data (i.e. a measure of ‘liking’), qualitative data and semantic differential data was collected. It was found that customers are clearly able to differentiate between switches based on the degree of liking for the samples' perceived haptic qualities, and that the assessment environment had a statistically significant effect, but that it was not universal. A factor analysis has shown that perceived characteristics of switch haptics can be explained by three independent factors defined as ‘Image’, ‘Build Quality’, and ‘Clickiness’. Preliminary steps have also been taken towards identifying whether existing theoretical frameworks for user experience may be applicable to automotive human–machine interfaces

Quality of Information in Mobile Crowdsensing: Survey and Research Challenges

Smartphones have become the most pervasive devices in people's lives, and are clearly transforming the way we live and perceive technology. Today's smartphones benefit from almost ubiquitous Internet connectivity and come equipped with a plethora of inexpensive yet powerful embedded sensors, such as accelerometer, gyroscope, microphone, and camera. This unique combination has enabled revolutionary applications based on the mobile crowdsensing paradigm, such as real-time road traffic monitoring, air and noise pollution, crime control, and wildlife monitoring, just to name a few. Differently from prior sensing paradigms, humans are now the primary actors of the sensing process, since they become fundamental in retrieving reliable and up-to-date information about the event being monitored. As humans may behave unreliably or maliciously, assessing and guaranteeing Quality of Information (QoI) becomes more important than ever. In this paper, we provide a new framework for defining and enforcing the QoI in mobile crowdsensing, and analyze in depth the current state-of-the-art on the topic. We also outline novel research challenges, along with possible directions of future work.Comment: To appear in ACM Transactions on Sensor Networks (TOSN

A legal and philosophical inquiry into affirmative action

This thesis explores the controversial question of affirmative action in higher education. The United States Supreme Court\u27s recent ruling in Grutter v. Bollinger, 539 U.S.---(2003) and Gratz v. Bollinger, 529 U.S.---(2003) is sparking a resurgence of debate over the issue. Both Grutter and Gratz filed lawsuits claiming that the University of Michigan affirmative action plan violated their right to equal protection of the laws because it served as a form of reverse discrimination by considering race in the admissions process. While the University of Michigan defended its use of affirmative action in higher education by citing the need for a diverse campus which originated in Justice Powell\u27s opinion in Regents of the University of California v. Bakke, 438 U.S. 265 (1978). The Supreme Court\u27s recent decision in these two cases has changed the nature of affirmative action making it a policy that no longer focuses solely on race but instead, looks at a variety of factors when deciding which applicants to admit

Customer perception of switch-feel in luxury sports utility vehicles

Successful new product introduction requires that product characteristics relate to the customer on functional, emotional, aesthetic and cultural levels. As a part of research into automotive human machine interfaces (HMI), this paper describes holistic customer research carried out to investigate how the haptics of switches in luxury sports utility vehicles (SUVs) are perceived by customers. The application of these techniques, including an initial proposal for objective specifications, is addressed within the broader new product introduction context, and benefits described. One-hundred and one customers of SUVs assessed the feel of automotive push switches, completing the tasks both in, and out of vehicles to investigate the effect of context. Using the semantic differential technique, hedonic testing, and content analysis of customers’ verbatim comments, a holistic picture has been built up of what influences the haptic experience. It was found that customers were able to partially discriminate differences in switch-feel, alongside considerations of visual appearance, image, and usability. Three factors named ‘Affective’, ‘Robustness and Precision’, and ‘Silkiness’ explained 61% of the variance in a principle components analysis. Correlations of the factors with acceptance scores were 0.505, 0.371, and 0.168, respectively

Describing Subjective Experiment Consistency by pp-Value P-P Plot

There are phenomena that cannot be measured without subjective testing. However, subjective testing is a complex issue with many influencing factors. These interplay to yield either precise or incorrect results. Researchers require a tool to classify results of subjective experiment as either consistent or inconsistent. This is necessary in order to decide whether to treat the gathered scores as quality ground truth data. Knowing if subjective scores can be trusted is key to drawing valid conclusions and building functional tools based on those scores (e.g., algorithms assessing the perceived quality of multimedia materials). We provide a tool to classify subjective experiment (and all its results) as either consistent or inconsistent. Additionally, the tool identifies stimuli having irregular score distribution. The approach is based on treating subjective scores as a random variable coming from the discrete Generalized Score Distribution (GSD). The GSD, in combination with a bootstrapped G-test of goodness-of-fit, allows to construct $p$-value P-P plot that visualizes experiment's consistency. The tool safeguards researchers from using inconsistent subjective data. In this way, it makes sure that conclusions they draw and tools they build are more precise and trustworthy. The proposed approach works in line with expectations drawn solely on experiment design descriptions of 21 real-life multimedia quality subjective experiments.Comment: 11 pages, 3 figures. Accepted to 28th ACM International Conference on Multimedia (MM '20). For associated data sets, source codes and documentation, see https://github.com/Qub3k/subjective-exp-consistency-chec

Construction contract policy: do we mean what we say?

Recent developments in contracting practice in the UK have built upon recommendations contained in highprofile reports, such as those by Latham and Egan. However, the New Engineering Contract (NEC), endorsed by Latham, is based upon principles of contract drafting that seem open to question. Any contract operates in the context of its legislative environment and current working practices. This report identifies eight contentious hypotheses in the literature on construction contracts and tests their validity in a sample survey that attracted 190 responses. The survey shows, among other things, that while partnership is a positive and useful idea, authoritative contract management is considered more effective and that “win-win” contracts, while desirable, are basically impractical. Further, precision and fairness in contracts are not easy to achieve simultaneously. While participants should know what is in their contracts, they should not routinely resort to legal action; and standard-form contracts should not seek to be universally applicable. Fundamental changes to drafting policy should be undertaken within the context of current legal contract doctrine and with a sensitivity to the way that contracts are used in contemporary practice. Attitudes to construction contracting may seem to be changing on the surface, but detailed analysis of what lies behind apparent agreement on new ways of working reveals that attitudes are changing much more slowly than they appear to be

Training the DNN of a Single Observer by Conducting Individualized Subjective Experiments

Predicting the quality perception of an individual subject instead of the mean opinion score is a new and very promising research direction. Deep Neural Networks (DNNs) are suitable for such prediction but the training process is particularly data demanding due to the noisy nature of individual opinion scores. We propose a human-in-the-loop training process using multiple cycles of a human voting, DNN training, and inference procedure. Thus, opinion scores on individualized sets of images were progressively collected from each observer to refine the performance of their DNN. The results of computational experiments demonstrate the effectiveness of our approach. For future research and benchmarking, five DNNs trained to mimic five observers are released together with a dataset containing the 1500 opinion scores progressively gathered from each of these observers during our training cycles

Determining the change in welfare estimates from introducing measurement error in non-linear choice models

Observed and unobserved characteristics of an individual are often used by researchers to explain choices over the provision of environmental goods. One means for identifying what is typically an unobserved characteristic, such as an attitude, is through some data reduction technique, such as factor analysis. However, the resultant variable represents the true attitude with measurement error, and hence, when included into a non-linear choice model, introduces bias in the model. There are well established methods to overcome this issue, which are seldom implemented. In an application to preferences over two water source alternatives for Perth in Western Australia, we use structural equation modeling within a discrete choice model to determine whether welfare measures are significantly impacted by ignoring measurement error in latent attitudes, and the advantage to policy makers from understanding what drives certain attitudes.contingent valuation, attitudes, structural equation modeling, recycled water, Environmental Economics and Policy, Research Methods/ Statistical Methods, Q51, Q53, C13,