14,740 research outputs found
Identifying Relationships between Physiological Measures and Evaluation Metrics for 3D Interaction Techniques
Abstract. This project aims to present a methodology to study the relationships between physiological measures and evaluation metrics for 3D interaction techniques using methods for multivariate data analysis. Physiological responses, such as heart rate and skin conductance, offer objective data about the user stress during interaction. This could be useful, for instance, to evaluate qualitative aspects of interaction techniques without relying on solely subjective data. Moreover, these data could contribute to improve task performance analysis by measuring different responses to 3D interaction techniques. With this in mind, we propose a methodology that defines a testing protocol, a normalization procedure and statistical techniques, considering the use of physiological measures during the evaluation process. A case study comparison between two 3D interaction techniques (ray-casting and HOMER) shows promising results, pointing to heart rate variability, as measured by the NN50 parameter, as a potential index of task performance. Further studies are needed in order to establish guidelines for evaluation processes based on welldefined associations between human behaviors and human actions realized in 3D user interfaces
Selecting Metrics to Evaluate Human Supervisory Control Applications
The goal of this research is to develop a methodology to select supervisory control metrics. This
methodology is based on cost-benefit analyses and generic metric classes. In the context of this research,
a metric class is defined as the set of metrics that quantify a certain aspect or component of a system.
Generic metric classes are developed because metrics are mission-specific, but metric classes are
generalizable across different missions. Cost-benefit analyses are utilized because each metric set has
advantages, limitations, and costs, thus the added value of different sets for a given context can be
calculated to select the set that maximizes value and minimizes costs. This report summarizes the
findings of the first part of this research effort that has focused on developing a supervisory control metric
taxonomy that defines generic metric classes and categorizes existing metrics. Future research will focus
on applying cost benefit analysis methodologies to metric selection.
Five main metric classes have been identified that apply to supervisory control teams composed
of humans and autonomous platforms: mission effectiveness, autonomous platform behavior efficiency,
human behavior efficiency, human behavior precursors, and collaborative metrics. Mission effectiveness
measures how well the mission goals are achieved. Autonomous platform and human behavior efficiency
measure the actions and decisions made by the humans and the automation that compose the team.
Human behavior precursors measure human initial state, including certain attitudes and cognitive
constructs that can be the cause of and drive a given behavior. Collaborative metrics address three
different aspects of collaboration: collaboration between the human and the autonomous platform he is
controlling, collaboration among humans that compose the team, and autonomous collaboration among
platforms. These five metric classes have been populated with metrics and measuring techniques from
the existing literature.
Which specific metrics should be used to evaluate a system will depend on many factors, but as a
rule-of-thumb, we propose that at a minimum, one metric from each class should be used to provide a
multi-dimensional assessment of the human-automation team. To determine what the impact on our
research has been by not following such a principled approach, we evaluated recent large-scale
supervisory control experiments conducted in the MIT Humans and Automation Laboratory. The results
show that prior to adapting this metric classification approach, we were fairly consistent in measuring
mission effectiveness and human behavior through such metrics as reaction times and decision
accuracies. However, despite our supervisory control focus, we were remiss in gathering attention
allocation metrics and collaboration metrics, and we often gathered too many correlated metrics that were
redundant and wasteful. This meta-analysis of our experimental shortcomings reflect those in the general
research population in that we tended to gravitate to popular metrics that are relatively easy to gather,
without a clear understanding of exactly what aspect of the systems we were measuring and how the
various metrics informed an overall research question
Stereoscopic video quality assessment using binocular energy
Stereoscopic imaging is becoming increasingly popular. However, to ensure the best quality of experience, there is a need to develop more robust and accurate objective metrics for stereoscopic content quality assessment. Existing stereoscopic image and video metrics are either extensions of conventional 2D
metrics (with added depth or disparity information) or are based on relatively simple perceptual models. Consequently, they tend to lack the accuracy and robustness required for stereoscopic content quality assessment. This paper introduces full-reference stereoscopic image and video quality metrics based on a Human
Visual System (HVS) model incorporating important physiological findings on binocular vision. The proposed approach is based on the following three contributions. First, it introduces a novel HVS model extending previous models to include the phenomena of binocular suppression and recurrent excitation. Second, an image quality metric based on the novel HVS model
is proposed. Finally, an optimised temporal pooling strategy is introduced to extend the metric to the video domain. Both image and video quality metrics are obtained via a training procedure to establish a relationship between subjective scores and objective measures of the HVS model. The metrics are evaluated using
publicly available stereoscopic image/video databases as well as a new stereoscopic video database. An extensive experimental evaluation demonstrates the robustness of the proposed quality metrics. This indicates a considerable improvement with respect to the state-of-the-art with average correlations with subjective
scores of 0.86 for the proposed stereoscopic image metric and 0.89 and 0.91 for the proposed stereoscopic video metrics
Full-reference stereoscopic video quality assessment using a motion sensitive HVS model
Stereoscopic video quality assessment has become a major research topic in recent years. Existing stereoscopic video quality metrics are predominantly based on stereoscopic image quality metrics extended to the time domain via for example temporal pooling. These approaches do not explicitly consider the motion sensitivity of the Human Visual System (HVS). To address this limitation, this paper introduces a novel HVS model inspired by physiological findings characterising the motion sensitive response of complex cells in the primary visual cortex (V1 area). The proposed HVS model generalises previous HVS models, which characterised the behaviour of simple and complex cells but ignored motion sensitivity, by estimating optical flow to measure scene velocity at different scales and orientations. The local motion characteristics (direction and amplitude) are used to modulate the output of complex cells. The model is applied to develop a new type of full-reference stereoscopic video quality metrics which uniquely combine non-motion sensitive and motion sensitive energy terms to mimic the response of the HVS. A tailored two-stage multi-variate stepwise regression algorithm is introduced to determine the optimal contribution of each energy term. The two proposed stereoscopic video quality metrics are evaluated on three stereoscopic video datasets. Results indicate that they achieve average correlations with subjective scores of 0.9257 (PLCC), 0.9338 and 0.9120 (SRCC), 0.8622 and 0.8306 (KRCC), and outperform previous stereoscopic video quality metrics including other recent HVS-based metrics
Biometric storyboards: a games user research approach for improving qualitative evaluations of player experience
Developing video games is an iterative and demanding process. It is difficult to achieve the goal of most video games â to be enjoyable, engaging and to create revenue for game developers â because of many hard-to-evaluate factors, such as the different ways players can interact with the game. Understanding how players behave during gameplay is of vital importance to developers and can be uncovered in user tests as part of game development. This can help developers to identify and resolve any potential problem areas before release, leading to a better player experience and possibly higher game review scores and sales. However, traditional user testing methods were developed for function and efficiency oriented applications. Hence, many traditional user testing methods cannot be applied in the same way for video game evaluation.
This thesis presents an investigation into the contributions of physiological measurements in user testing within games user research (GUR). GUR specifically studies the interaction between a game and users (players) with the aim to provide feedback for developers to help them to optimise the game design of their title. An evaluation technique called Biometric Storyboards is developed, which visualises the relationships between game events, player
feedback and changes in a playerâs physiological state. Biometric Storyboards contributes to the field of human-computer interaction and GUR in three important areas: (1) visualising mixedmeasures of player experience, (2) deconstructing game design by analysing game events and pace, (3) incremental improvement of classic user research techniques (such as interviews and physiological measurements).
These contributions are described in practical case studies, interviews with game developers and laboratory experiments. The results show this evaluation approach can enable games user researchers to increase the plausibility and persuasiveness of their reports and facilitate developers to better deliver their design goals. Biometric Storyboards is not aimed at replacing existing methods, but to extend them with mixed methods visualisations, to provide powerful tools for games user researchers and developers to better understand and communicate player needs, interactions and experiences. The contributions of this thesis are directly applicable for user researchers and game developers, as well as for researchers in user experience evaluation in entertainment systems
- âŠ