Transformation of Mean Opinion Scores to Avoid Misleading of Ranked based Statistical Techniques

The rank correlation coefficients and the ranked-based statistical tests (as a subset of non-parametric techniques) might be misleading when they are applied to subjectively collected opinion scores. Those techniques assume that the data is measured at least at an ordinal level and define a sequence of scores to represent a tied rank when they have precisely an equal numeric value. In this paper, we show that the definition of tied rank, as mentioned above, is not suitable for Mean Opinion Scores (MOS) and might be misleading conclusions of rank-based statistical techniques. Furthermore, we introduce a method to overcome this issue by transforming the MOS values considering their $95\%$ Confidence Intervals. The rank correlation coefficients and ranked-based statistical tests can then be safely applied to the transformed values. We also provide open-source software packages in different programming languages to utilize the application of our transformation method in the quality of experience domain.Comment: his paper has been accepted for publication in the 2020 Twelfth International Conference on Quality of Multimedia Experience (QoMEX

Application of Just-Noticeable Difference in Quality as Environment Suitability Test for Crowdsourcing Speech Quality Assessment Task

Crowdsourcing micro-task platforms facilitate subjective media quality assessment by providing access to a highly scale-able, geographically distributed and demographically diverse pool of crowd workers. Those workers participate in the experiment remotely from their own working environment, using their own hardware. In the case of speech quality assessment, preliminary work showed that environmental noise at the listener's side and the listening device (loudspeaker or headphone) significantly affect perceived quality, and consequently the reliability and validity of subjective ratings. As a consequence, ITU-T Rec. P.808 specifies requirements for the listening environment of crowd workers when assessing speech quality. In this paper, we propose a new Just Noticeable Difference of Quality (JNDQ) test as a remote screening method for assessing the suitability of the work environment for participating in speech quality assessment tasks. In a laboratory experiment, participants performed this JNDQ test with different listening devices in different listening environments, including a silent room according to ITU-T Rec. P.800 and a simulated background noise scenario. Results show a significant impact of the environment and the listening device on the JNDQ threshold. Thus, the combination of listening device and background noise needs to be screened in a crowdsourcing speech quality test. We propose a minimum threshold of our JNDQ test as an easily applicable screening method for this purpose.Comment: This paper has been accepted for publication in the 2020 Twelfth International Conference on Quality of Multimedia Experience (QoMEX

Towards a new ITU-T recommendation for subjective methods evaluating gaming QoE

This paper reports on activities in Study Group 12 of the International Telecommunication Union (ITU-T SG12) to define a new Recommendation on subjective evaluation methods for gaming Quality of Experience (QoE). It first resumes the structure and content of the current draft which has been proposed to ITU-T SG12 in September 2014 and then critically discusses potential gaming content and evaluation methods for inclusion into the upcoming Recommendation. The aim is to start a discussion amongst experts on potential evaluation methods and their limitations, before finalizing a Recommendation. Such a recommendation might in the end be applied by non -expert users, hence wrong decisions in the evaluation design could negatively affect gaming QoE throughout the evaluation

Psychological needs as motivators for security and privacy actions on smartphones

Much work has been conducted to investigate the obstacles that keep users from using mitigations against security and privacy threats on smartphones. By contrast, we conducted in-depth interviews (N = 19) to explore users’ motivations for voluntarily applying security and privacy actions on smartphones. Our work focuses on analyzing intrinsic motivation in terms of psychological need fulfillment. The findings from the interview study provide first insights on the salience of basic psychological needs in the context of smartphone security and privacy. They illustrate how security and privacy actions on smartphones are motivated by a variety of psychological needs, only one of them being the need for Security. We further conducted an online survey (N = 70) in which we used questionnaires on psychological need fulfillment from the literature. The online survey is a first attempt to quantify psychological need fulfillment for security and privacy actions on smartphones. Whereas the results of the interview study indicate that Security and other needs play a role as motivators for employing security and privacy actions on smartphones, the online study does not support the need for Security as an outstanding motivator. Instead, in the online study, other needs such as Keeping the meaningful, Stimulation, Autonomy, and Competence show to be rather salient as motivators for security and privacy actions. Furthermore, the mean need fulfillment for security and privacy actions is in general rather low in the online survey. We conclude that there is scope for improvement to maximize psychological need fulfillment with security and privacy actions. In order to achieve a positive user experience with security and privacy technologies on smartphones, we suggest addressing additional psychological needs, beyond the need for Security, in the design of such technologies