Beyond blur: real-time ventral metamers for foveated rendering

To peripheral vision, a pair of physically different images can look the same. Such pairs are metamers relative to each other, just as physically-different spectra of light are perceived as the same color. We propose a real-time method to compute such ventral metamers for foveated rendering where, in particular for near-eye displays, the largest part of the framebuffer maps to the periphery. This improves in quality over state-of-the-art foveation methods which blur the periphery. Work in Vision Science has established how peripheral stimuli are ventral metamers if their statistics are similar. Existing methods, however, require a costly optimization process to find such metamers. To this end, we propose a novel type of statistics particularly well-suited for practical real-time rendering: smooth moments of steerable filter responses. These can be extracted from images in time constant in the number of pixels and in parallel over all pixels using a GPU. Further, we show that they can be compressed effectively and transmitted at low bandwidth. Finally, computing realizations of those statistics can again be performed in constant time and in parallel. This enables a new level of quality for foveated applications such as such as remote rendering, level-of-detail and Monte-Carlo denoising. In a user study, we finally show how human task performance increases and foveation artifacts are less suspicious, when using our method compared to common blurring

Perceptual models for high-refresh-rate rendering

Rendering realistic images requires substantial computational power. With new high-refresh-rate displays as well as the renaissance of virtual reality (VR) and augmented reality (AR), one cannot expect that GPU performance will scale fast enough to meet the requirements of immersive photo-realistic rendering with current rendering techniques. In this dissertation, I follow the dual of the well-known computer vision approach: vision is inverse graphics: to improve graphical algorithms, I consider the operation of the human visual system. I propose to model and exploit the limitations of the visual system in the context of novel high-refresh-rate displays; specifically, I focus on spatio-temporal perception, a topic that has received remarkably less attention than spatial-only perception so far. I present three main contributions. First, I demonstrate the validity of the perceptual approach by presenting a conceptually simple rendering technique motivated by our eyes' limited sensitivity to high spatio-temporal change which reduces the rendering load and transmission requirement of current-generation VR headsets without introducing perceivable visual artefacts. Second, I present two visual models related to motion perception: (a) a metric for detecting flicker; and (b) a comprehensive visual model to predict perceived motion quality on monitors with arbitrary refresh rates and monitor resolutions. Third, I propose an adaptive rendering algorithm that utilises the proposed models. All algorithms operate on physical colorimetric units (instead of display-referenced pixel values), for which I provide the appropriate display measurements and models. All proposed algorithms and visual models are calibrated and validated with psychophysical experiments

Eye-Tracking-Based Classification of Information Search Behavior Using Machine Learning: Evidence from Experiments in Physical Shops and Virtual Reality Shopping Environments

Classifying information search behavior helps tailor recommender systems to individual customers’ shopping motives. But how can we identify these motives without requiring users to exert too much effort? Our research goal is to demonstrate that eye tracking can be used at the point of sale to do so. We focus on two frequently investigated shopping motives: goal-directed and exploratory search. To train and test a prediction model, we conducted two eye-tracking experiments in front of supermarket shelves. The first experiment was carried out in immersive virtual reality; the second, in physical reality—in other words, as a field study in a real supermarket. We conducted a virtual reality study, because recently launched virtual shopping environments suggest that there is great interest in using this technology as a retail channel. Our empirical results show that support vector machines allow the correct classification of search motives with 80% accuracy in virtual reality and 85% accuracy in physical reality. Our findings also imply that eye movements allow shopping motives to be identified relatively early in the search process: our models achieve 70% prediction accuracy after only 15 seconds in virtual reality and 75% in physical reality. Applying an ensemble method increases the prediction accuracy substantially, to about 90%. Consequently, the approach that we propose could be used for the satisfiable classification of consumers in practice. Furthermore, both environments’ best predictor variables overlap substantially. This finding provides evidence that in virtual reality, information search behavior might be similar to the one used in physical reality. Finally, we also discuss managerial implications for retailers and companies that are planning to use our technology to personalize a consumer assistance system

Perceptual model for adaptive local shading and refresh rate

When the rendering budget is limited by power or time, it is necessary to find the combination of rendering parameters, such as resolution and refresh rate, that could deliver the best quality. Variable-rate shading (VRS), introduced in the last generations of GPUs, enables fine control of the rendering quality, in which each 16×16 image tile can be rendered with a different ratio of shader executions. We take advantage of this capability and propose a new method for adaptive control of local shading and refresh rate. The method analyzes texture content, on-screen velocities, luminance, and effective resolution and suggests the refresh rate and a VRS state map that maximizes the quality of animated content under a limited budget. The method is based on the new content-adaptive metric of judder, aliasing, and blur, which is derived from the psychophysical models of contrast sensitivity. To calibrate and validate the metric, we gather data from literature and also collect new measurements of motion quality under variable shading rates, different velocities of motion, texture content, and display capabilities, such as refresh rate, persistence, and angular resolution. The proposed metric and adaptive shading method is implemented as a game engine plugin. Our experimental validation shows a substantial increase in preference of our method over rendering with a fixed resolution and refresh rate, and an existing motion-adaptive techniqu

Design guidelines for limiting and eliminating virtual reality-induced symptoms and effects at work: a comprehensive, factor-oriented review

Virtual reality (VR) can induce side effects known as virtual reality-induced symptoms and effects (VRISE). To address this concern, we identify a literature-based listing of these factors thought to influence VRISE with a focus on office work use. Using those, we recommend guidelines for VRISE amelioration intended for virtual environment creators and users. We identify five VRISE risks, focusing on short-term symptoms with their short-term effects. Three overall factor categories are considered: individual, hardware, and software. Over 90 factors may influence VRISE frequency and severity. We identify guidelines for each factor to help reduce VR side effects. To better reflect our confidence in those guidelines, we graded each with a level of evidence rating. Common factors occasionally influence different forms of VRISE. This can lead to confusion in the literature. General guidelines for using VR at work involve worker adaptation, such as limiting immersion times to between 20 and 30 min. These regimens involve taking regular breaks. Extra care is required for workers with special needs, neurodiversity, and gerontechnological concerns. In addition to following our guidelines, stakeholders should be aware that current head-mounted displays and virtual environments can continue to induce VRISE. While no single existing method fully alleviates VRISE, workers' health and safety must be monitored and safeguarded when VR is used at work

Content-prioritised video coding for British Sign Language communication.

Video communication of British Sign Language (BSL) is important for remote interpersonal communication and for the equal provision of services for deaf people. However, the use of video telephony and video conferencing applications for BSL communication is limited by inadequate video quality. BSL is a highly structured, linguistically complete, natural language system that expresses vocabulary and grammar visually and spatially using a complex combination of facial expressions (such as eyebrow movements, eye blinks and mouth/lip shapes), hand gestures, body movements and finger-spelling that change in space and time. Accurate natural BSL communication places specific demands on visual media applications which must compress video image data for efficient transmission. Current video compression schemes apply methods to reduce statistical redundancy and perceptual irrelevance in video image data based on a general model of Human Visual System (HVS) sensitivities. This thesis presents novel video image coding methods developed to achieve the conflicting requirements for high image quality and efficient coding. Novel methods of prioritising visually important video image content for optimised video coding are developed to exploit the HVS spatial and temporal response mechanisms of BSL users (determined by Eye Movement Tracking) and the characteristics of BSL video image content. The methods implement an accurate model of HVS foveation, applied in the spatial and temporal domains, at the pre-processing stage of a current standard-based system (H.264). Comparison of the performance of the developed and standard coding systems, using methods of video quality evaluation developed for this thesis, demonstrates improved perceived quality at low bit rates. BSL users, broadcasters and service providers benefit from the perception of high quality video over a range of available transmission bandwidths. The research community benefits from a new approach to video coding optimisation and better understanding of the communication needs of deaf people

Near Transfer After Direct Instruction: An Experimental Inquiry within Aviation Technician Training

This study put forth two instructional interventions set within a direct instruction (DI) framework specific to an aviation maintenance context. To evaluate the effectiveness of these two training interventions a criterion was established to measure near transfer during a performance evaluation on a live aircraft. Information learned within this study indicates that DI can be highly effective in technical training environments. This study also articulates how VR experiences may be included within these types of training contexts and discusses the factors and affordances that come with utilizing VR in instructional activities. Additionally, this study revealed experiential characteristics of a DI training experience from the learner perspective. Most notable among them was how much emphasis learners placed on the Present phase of the direct instruction framework, oftentimes discussing the quality, usefulness, and preference of the study’s training videos comparative to other forms of instructional media, including even the study’s VR experience itself. Finally, this study leveraged a novel research design for both the instructional context and the study’s unit of measurement in near transfer. This study exemplifies how within-subject repeated measure design may be an ideal framework for researchers looking to address long-standing critiques of experimental research within the field of instructional design

Online mouse tracking as a measure of attention in videos, using a mouse-contingent bi-resolution display

Master of ScienceDepartment of Psychological SciencesLester C LoschkyData on human visual attention is increasingly collected online, but there are limited tools available to study attention to video stimuli in online experiments. Webcam-based eye tracking is improving, but it faces issues with precision and attrition that prevent its adoption by many researchers. Here I detail an alternative mouse-based paradigm that can be used to measure attention to videos online. This method uses a blurred display and a high-resolution window centered on the user’s computer mouse location. As the user moves their mouse to view different screen content, their mouse movements are recorded, providing an approximation of eye movements and the attended screen location. To validate this Mouse-Contingent Bi-Resolution Display (MCBRD) paradigm, mouse movements collected from online participants watching twenty-seven videos were compared to eye movements from the DIEM dataset. Display settings of window size and blur level were manipulated to identify the settings that resulted in mouse movements most similar to eye movements. This validation study found differences in speed between mouse and eye movements, but similarities in attended regions of interest, especially when the MCBRD screen was blurred with the highest tested Gaussian blur sigma of 0.45 degrees of visual angle. These results suggest that the MCBRD paradigm can be used to measure what regions viewers find salient, interesting, or visually informative in online videos

The challenges of developing a contrast-based video game for treatment of amblyopia

Perceptual learning of visual tasks is emerging as a promising treatment for amblyopia, a developmental disorder of vision characterized by poor monocular visual acuity. The tasks tested thus far span the gamut from basic psychophysical discriminations to visually complex video games. One end of the spectrum offers precise control over stimulus parameters, whilst the other delivers the benefits of motivation and reward that sustain practice over long periods. Here, we combined the advantages of both approaches by developing a video game that trains contrast sensitivity, which in psychophysical experiments, is associated with significant improvements in visual acuity in amblyopia. Target contrast was varied adaptively in the game to derive a contrast threshold for each session. We tested the game on 20 amblyopic subjects (10 children and 10 adults), who played at home using their amblyopic eye for an average of 37 sessions (approximately 11 h). Contrast thresholds from the game improved reliably for adults but not for children. However, logMAR acuity improved for both groups (mean = 1.3 lines; range = 0–3.6 lines). We present the rationale leading to the development of the game and describe the challenges of incorporating psychophysical methods into game-like settings