A perceptual model of motion quality for rendering with adaptive refresh-rate and resolution

Limited GPU performance budgets and transmission bandwidths mean that real-time rendering often has to compromise on the spatial resolution or temporal resolution (refresh rate). A common practice is to keep either the resolution or the refresh rate constant and dynamically control the other variable. But this strategy is non-optimal when the velocity of displayed content varies. To find the best trade-off between the spatial resolution and refresh rate, we propose a perceptual visual model that predicts the quality of motion given an object velocity and predictability of motion. The model considers two motion artifacts to establish an overall quality score: non-smooth (juddery) motion, and blur. Blur is modeled as a combined effect of eye motion, finite refresh rate and display resolution. To fit the free parameters of the proposed visual model, we measured eye movement for predictable and unpredictable motion, and conducted psychophysical experiments to measure the quality of motion from 50 Hz to 165 Hz. We demonstrate the utility of the model with our on-the-fly motion-adaptive rendering algorithm that adjusts the refresh rate of a G-Sync-capable monitor based on a given rendering budget and observed object motion. Our psychophysical validation experiments demonstrate that the proposed algorithm performs better than constant-refresh-rate solutions, showing that motion-adaptive rendering is an attractive technique for driving variable-refresh-rate displays.</jats:p

Speaking Rate Effects on Locus Equation Slope

A locus equation describes a 1st order regression fit to a scatter of vowel steady-state frequency values predicting vowel onset frequency values. Locus equation coefficients are often interpreted as indices of coarticulation. Speaking rate variations with a constant consonant–vowel form are thought to induce changes in the degree of coarticulation. In the current work, the hypothesis that locus slope is a transparent index of coarticulation is examined through the analysis of acoustic samples of large-scale, nearly continuous variations in speaking rate. Following the methodological conventions for locus equation derivation, data pooled across ten vowels yield locus equation slopes that are mostly consistent with the hypothesis that locus equations vary systematically with coarticulation. Comparable analyses between different four-vowel pools reveal variations in the locus slope range and changes in locus slope sensitivity to rate change. Analyses across rate but within vowels are substantially less consistent with the locus hypothesis. Taken together, these findings suggest that the practice of vowel pooling exerts a non-negligible influence on locus outcomes. Results are discussed within the context of articulatory accounts of locus equations and the effects of speaking rate change

Effects on orientation perception of manipulating the spatio–temporal prior probability of stimuli

Spatial and temporal regularities commonly exist in natural visual scenes. The knowledge of the probability structure of these regularities is likely to be informative for an efficient visual system. Here we explored how manipulating the spatio–temporal prior probability of stimuli affects human orientation perception. Stimulus sequences comprised four collinear bars (predictors) which appeared successively towards the foveal region, followed by a target bar with the same or different orientation. Subjects' orientation perception of the foveal target was biased towards the orientation of the predictors when presented in a highly ordered and predictable sequence. The discrimination thresholds were significantly elevated in proportion to increasing prior probabilities of the predictors. Breaking this sequence, by randomising presentation order or presentation duration, decreased the thresholds. These psychophysical observations are consistent with a Bayesian model, suggesting that a predictable spatio–temporal stimulus structure and an increased probability of collinear trials are associated with the increasing prior expectation of collinear events. Our results suggest that statistical spatio–temporal stimulus regularities are effectively integrated by human visual cortex over a range of spatial and temporal positions, thereby systematically affecting perception

Perceptual models for high-refresh-rate rendering

Rendering realistic images requires substantial computational power. With new high-refresh-rate displays as well as the renaissance of virtual reality (VR) and augmented reality (AR), one cannot expect that GPU performance will scale fast enough to meet the requirements of immersive photo-realistic rendering with current rendering techniques. In this dissertation, I follow the dual of the well-known computer vision approach: vision is inverse graphics: to improve graphical algorithms, I consider the operation of the human visual system. I propose to model and exploit the limitations of the visual system in the context of novel high-refresh-rate displays; specifically, I focus on spatio-temporal perception, a topic that has received remarkably less attention than spatial-only perception so far. I present three main contributions. First, I demonstrate the validity of the perceptual approach by presenting a conceptually simple rendering technique motivated by our eyes' limited sensitivity to high spatio-temporal change which reduces the rendering load and transmission requirement of current-generation VR headsets without introducing perceivable visual artefacts. Second, I present two visual models related to motion perception: (a) a metric for detecting flicker; and (b) a comprehensive visual model to predict perceived motion quality on monitors with arbitrary refresh rates and monitor resolutions. Third, I propose an adaptive rendering algorithm that utilises the proposed models. All algorithms operate on physical colorimetric units (instead of display-referenced pixel values), for which I provide the appropriate display measurements and models. All proposed algorithms and visual models are calibrated and validated with psychophysical experiments

How spatial constraints on efficacy and dynamic signaling alignment shape animal communication

Effective communication is important to the survival and reproduction of many organisms. Signal transmission and reception have spatial constraints that interact to determine effectiveness. Signals are often best perceived from specific angles, and sensory systems may be limited in their ability to detect or interpret incoming stimuli from certain directions. Alignment between these directional biases can be critical to effective communication. Misalignment of either signal or sensor may disrupt signal perception. Signals also degrade during the distance traveled from signaler to receiver. Thus, how animals position themselves during communication may be under selection. Despite this, we know little about the spatial arrangement of signalers and receivers, what behaviors influence positioning, or the causes and consequences of variation in positioning. To address this fundamental gap in knowledge, I developed a geometric framework for studying the spatial constraints of communication and how they shape positioning across visual, sound, and chemical signaling. To investigate respective roles of signaler and receiver in managing these spatial constraints, I then characterized the spatial dynamics of visual signaling in the jumping spider Habronattus pyrrithrix. Males perform an elaborate courtship dance which includes arm waves and colorful ornaments; the latter are not visible from the side. The female can only perceive colors of male displays when they are presented in her frontal field of view. I recorded relative positions and orientations of both actors throughout courtship and established the role of each sex in maintaining signaling alignment. Finally, I tested how males control female orientation, and respond to signaling challenges. Using video playback, I asked how males’ arm-waving display and the visual environment determine how effectively they attracted female attention, as well as how males respond to variation in their signaling environment. These studies reveal that signaling alignment is frequently disrupted by females turning away from males, but that male arm-waving effectively re-captures her attention. Males also modulate displays in response to changing spatial and environmental conditions. Overall, this dissertation reveals the widespread role of spatial constraints in driving signaling behavior, and demonstrates that the spatial arrangement of signaler and receiver must be managed dynamically through behavioral responses

Perceptual modelling for 2D and 3D

Livrable D1.1 du projet ANR PERSEECe rapport a été réalisé dans le cadre du projet ANR PERSEE (n° ANR-09-BLAN-0170). Exactement il correspond au livrable D1.1 du projet

Perception of Color Break-Up

Hintergrund. Ein farbverfälschender Bildfehler namens Color Break-Up (CBU) wurde untersucht. Störende CBU-Effekte treten auf, wenn Augenbewegungen (z.B. Folgebewegungen oder Sakkaden) während der Content-Wiedergabe über sogenannte Field-Sequential Color (FSC) Displays oder Projektoren ausgeführt werden. Die Ursache für das Auftreten des CBU-Effektes ist die sequenzielle Anzeige der Primärfarben über das FSC-System. Methoden. Ein kombiniertes Design aus empirischer Forschung und theoretischer Modellierung wurde angewendet. Mittels empirischer Studien wurde der Einfluss von hardware-, content- und betrachterbasierten Faktoren auf die CBU-Wahrnehmung der Stichprobe untersucht. Hierzu wurden zunächst Sehleistung (u. a. Farbsehen), Kurzzeitzustand (u. a. Aufmerksamkeit) und Persönlichkeitsmerkmale (u. a. Technikaffinität) der Stichprobe erfasst. Anschließend wurden die Teilnehmenden gebeten, die wahrgenommene CBU-Intensität verschiedener Videosequenzen zu bewerten. Die Sequenzen wurden mit einem FSC-Projektor wiedergegeben. Das verwendete Setup ermöglichte die Untersuchung folgender Variablen: die Größe (1.0 bis 6.0°) und Leuchtdichte (10.0 bis 157.0 cd/m2) des CBU-provozierenden Contents, das Augenbewegungsmuster des Teilnehmenden (Geschwindigkeit der Folgebewegung: 18.0 bis 54.0 °/s; Amplitude der Sakkade: 3.6 bis 28.2°), die Position der Netzhautstimulation (0.0 bis 50.0°) und die Bildrate des Projektors (30.0 bis 420.0 Hz). Korrelationen zwischen den unabhängigen Variablen und der subjektiven CBU-Wahrnehmung wurden getestet. Das ergänzend entwickelte Modell prognostiziert die CBU-Wahrnehmung eines Betrachters auf theoretische Weise. Das Modell rekonstruiert die Intensitäts- und Farbeigenschaften von CBU-Effekten zunächst grafisch. Anschließend wird die visuelle CBU-Rekonstruktion zu repräsentativen Modellindizes komprimiert, um das modellierte Szenario mit einem handhabbaren Satz von Metriken zu quantifizieren. Die Modellergebnisse wurden abschließend mit den empirischen Daten verglichen. Ergebnisse. Die hohe interindividuelle CBU-Variabilität innerhalb der Stichprobe lässt sich nicht durch die Sehleistung, den Kurzzeitzustand oder die Persönlichkeitsmerkmale eines Teilnehmenden erklären. Eindeutig verstärkende Bedingungen der CBU-Wahrnehmung sind: (1) eine foveale Position des CBU-Stimulus, (2) eine reduzierte Stimulusgröße während Sakkaden, (3) eine hohe Bewegungsgeschwindigkeit des Auges und (4) eine niedrige Bildrate des Projektors (Korrelation durch Exponentialfunktion beschreibbar, r2 > .93). Die Leuchtdichte des Stimulus wirkt sich nur geringfügig auf die CBU-Wahrnehmung aus. Generell hilft das Modell, die grundlegenden Prozesse der CBU-Genese zu verstehen, den Einfluss von CBU-Determinanten zu untersuchen und ein Klassifizierungsschema für verschiedene CBU-Varianten zu erstellen. Das Modell prognostiziert die empirischen Daten innerhalb der angegebenen Toleranzbereiche. Schlussfolgerungen. Die Studienergebnisse ermöglichen die Festlegung von Bildraten und Eigenschaften des CBU-provozierenden Content (Größe und Position), die das Überschreiten vordefinierter, störender CBU-Grenzwerte vermeiden. Die abgeleiteten Hardwareanforderungen und Content-Empfehlungen ermöglichen ein praxisnahes und evidenzbasiertes CBU-Management. Für die Vorhersage von CBU kann die Modellgenauigkeit weiter verbessert werden, indem Merkmale der menschlichen Wahrnehmung berücksichtigt werden, z.B. die exzentrizitätsabhängige Netzhautempfindlichkeit oder Änderungen der visuellen Wahrnehmung bei unterschiedlichen Arten von Augenbewegungen. Zur Modellierung dieser Merkmale können teilnehmerbezogene Daten der empirischen Forschung herangezogen werden.Background. A color-distorting artifact called Color Break-Up (CBU) has been investigated. Disturbing CBU effects occur when eye movements (e.g., pursuits or saccades) are performed during the presentation of content on Field-Sequential Color (FSC) display or projection systems where the primary colors are displayed sequentially rather than simultaneously. Methods. A mixed design of empirical research and theoretical modeling was used to address the main research questions. Conducted studies evaluated the impact of hardware-based, content-based, and viewer-based factors on the sample’s CBU perception. In a first step, visual performance parameters (e.g., color vision), short-term state (e.g., attention level), and long-term personality traits (e.g., affinity to technology) of the sample were recorded. Participants were then asked to rate the perceived CBU intensity for different video sequences presented by a FSC-based projector. The applied setup allowed the size of the CBU-provoking content (1.0 to 6.0°), its luminance level (10.0 to 157.0 cd/m2), the participant’s eye movement pattern (pursuits: 18.0 to 54.0 deg/s; saccadic amplitudes: 3.6 to 28.2°), the position of retinal stimulation (0.0 to 50.0°), and the projector’s frame rate (30.0 to 420.0 Hz) to be varied. Correlations between independent variables and subjective CBU perception were tested. In contrast, the developed model predicts a viewer’s CBU perception on an objective basis. The model graphically reconstructs the intensity and color characteristics of CBU effects. The visual CBU reconstruction is then compressed into representative model indices to quantify the modeled scenario with a manageable set of metrics. Finally, the model output was compared to the empirical data. Results. High interindividual CBU variability within the sample cannot be explained by a participant’s visual performance, short-term state or long-term personality traits. Conditions that distinctly elevate the participant’s CBU perception are (1) a foveal stimulus position on the retina, (2) a small-sized stimulus during saccades, (3) a high eye movement velocity, and (4) a low frame rate of the projector (correlation expressed by exponential function, r2 > .93). The stimulus luminance, however, only slightly affects CBU perception. In general, the model helps to understand the fundamental processes of CBU genesis, to investigate the impact of CBU determinants, and to establish a classification scheme for different CBU variants. The model adequately predicts the empirical data within the specified tolerance ranges. Conclusions. The study results allow the determination of frame rates and content characteristics (size and position) that avoid exceeding predefined annoyance thresholds for CBU perception. The derived hardware requirements and content recommendations enable practical and evidence-based CBU management. For CBU prediction, model accuracy can be further improved by considering features of human perception, e.g., eccentricity-dependent retinal sensitivity or changes in visual perception with different types of eye movements. Participant-based data from the empirical research can be used to model these features

Forward Sensitivity Analysis and Mode Dependent Control for Closure Modeling of Galerkin Systems

Model reduction by projection-based approaches is often associated with losing some of the important features that contribute towards the dynamics of the retained scales. As a result, a mismatch occurs between the predicted trajectories of the original system and the truncated one. We put forth a framework to apply a continuous time control signal in the latent space of the reduced order model (ROM) to account for the effect of truncation. We set the control input using parameterized models by following energy transfer principles. Our methodology relies on observing the system behavior in the physical space and using the projection operator to restrict the feedback signal into the latent space. Then, we leverage the forward sensitivity method (FSM) to derive relationships between the feedback and the desired mode-dependent control. We test the performance of the proposed approach using two test cases, corresponding to viscous Burgers and vortex merger problems at high Reynolds number. Results show that the ROM trajectory with the applied FSM control closely matches its target values in both the data-dense and data-sparse regimes

Choosing Your Poison: Optimizing Simulator Visual System Selection as a Function of Operational Tasks

Although current technology simulator visual systems can achieve extremely realistic levels they do not completely replicate the experience of a pilot sitting in the cockpit, looking at the outside world. Some differences in experience are due to visual artifacts, or perceptual features that would not be present in a naturally viewed scene. Others are due to features that are missing from the simulated scene. In this paper, these differences will be defined and discussed. The significance of these differences will be examined as a function of several particular operational tasks. A framework to facilitate the choice of visual system characteristics based on operational task requirements will be proposed