Practical Saccade Prediction for Head-Mounted Displays: Towards a Comprehensive Model

Eye-tracking technology is an integral component of new display devices suchas virtual and augmented reality headsets. Applications of gaze informationrange from new interaction techniques exploiting eye patterns togaze-contingent digital content creation. However, system latency is still asignificant issue in many of these applications because it breaks thesynchronization between the current and measured gaze positions. Consequently,it may lead to unwanted visual artifacts and degradation of user experience. Inthis work, we focus on foveated rendering applications where the quality of animage is reduced towards the periphery for computational savings. In foveatedrendering, the presence of latency leads to delayed updates to the renderedframe, making the quality degradation visible to the user. To address thisissue and to combat system latency, recent work proposes to use saccade landingposition prediction to extrapolate the gaze information from delayedeye-tracking samples. While the benefits of such a strategy have already beendemonstrated, the solutions range from simple and efficient ones, which makeseveral assumptions about the saccadic eye movements, to more complex andcostly ones, which use machine learning techniques. Yet, it is unclear to whatextent the prediction can benefit from accounting for additional factors. Thispaper presents a series of experiments investigating the importance ofdifferent factors for saccades prediction in common virtual and augmentedreality applications. In particular, we investigate the effects of saccadeorientation in 3D space and smooth pursuit eye-motion (SPEM) and how theirinfluence compares to the variability across users. We also present a simpleyet efficient correction method that adapts the existing saccade predictionmethods to handle these factors without performing extensive data collection.<br

Motion Parallax in Stereo 3D: Model and Applications

Binocular disparity is the main depth cue that makes stereoscopic images appear 3D. However, in many scenarios, the range of depth that can be reproduced by this cue is greatly limited and typically fixed due to constraints imposed by displays. For example, due to the low angular resolution of current automultiscopic screens, they can only reproduce a shallow depth range. In this work, we study the motion parallax cue, which is a relatively strong depth cue, and can be freely reproduced even on a 2D screen without any limits. We exploit the fact that in many practical scenarios, motion parallax provides sufficiently strong depth information that the presence of binocular depth cues can be reduced through aggressive disparity compression. To assess the strength of the effect we conduct psycho-visual experiments that measure the influence of motion parallax on depth perception and relate it to the depth resulting from binocular disparity. Based on the measurements, we propose a joint disparity-parallax computational model that predicts apparent depth resulting from both cues. We demonstrate how this model can be applied in the context of stereo and multiscopic image processing, and propose new disparity manipulation techniques, which first quantify depth obtained from motion parallax, and then adjust binocular disparity information accordingly. This allows us to manipulate the disparity signal according to the strength of motion parallax to improve the overall depth reproduction. This technique is validated in additional experiments

Learning Foveated Reconstruction to Preserve Perceived Image Statistics

Foveated image reconstruction recovers full image from a sparse set of samples distributed according to the human visual system's retinal sensitivity that rapidly drops with eccentricity. Recently, the use of Generative Adversarial Networks was shown to be a promising solution for such a task as they can successfully hallucinate missing image information. Like for other supervised learning approaches, also for this one, the definition of the loss function and training strategy heavily influences the output quality. In this work, we pose the question of how to efficiently guide the training of foveated reconstruction techniques such that they are fully aware of the human visual system's capabilities and limitations, and therefore, reconstruct visually important image features. Due to the nature of GAN-based solutions, we concentrate on the human's sensitivity to hallucination for different input sample densities. We present new psychophysical experiments, a dataset, and a procedure for training foveated image reconstruction. The strategy provides flexibility to the generator network by penalizing only perceptually important deviations in the output. As a result, the method aims to preserve perceived image statistics rather than natural image statistics. We evaluate our strategy and compare it to alternative solutions using a newly trained objective metric and user experiments

Selecting texture resolution using a task-specific visibility metric

In real-time rendering, the appearance of scenes is greatly affected by the quality and resolution of the textures used for image synthesis. At the same time, the size of textures determines the performance and the memory requirements of rendering. As a result, finding the optimal texture resolution is critical, but also a non-trivial task since the visibility of texture imperfections depends on underlying geometry, illumination, interactions between several texture maps, and viewing positions. Ideally, we would like to automate the task with a visibility metric, which could predict the optimal texture resolution. To maximize the performance of such a metric, it should be trained on a given task. This, however, requires sufficient user data which is often difficult to obtain. To address this problem, we develop a procedure for training an image visibility metric for a specific task while reducing the effort required to collect new data. The procedure involves generating a large dataset using an existing visibility metric followed by refining that dataset with the help of an efficient perceptual experiment. Then, such a refined dataset is used to retune the metric. This way, we augment sparse perceptual data to a large number of per-pixel annotated visibility maps which serve as the training data for application-specific visibility metrics. While our approach is general and can be potentially applied for different image distortions, we demonstrate an application in a game-engine where we optimize the resolution of various textures, such as albedo and normal maps

Gloss Management for Consistent Reproduction of Real and Virtual Objects

A good match of material appearance between real-world objects and their digital on-screen representations is critical for many applications such as fabrication, design, and e-commerce. However, faithful appearance reproduction is challenging, especially for complex phenomena, such as gloss. In most cases, the view-dependent nature of gloss and the range of luminance values required for reproducing glossy materials exceeds the current capabilities of display devices. As a result, appearance reproduction poses significant problems even with accurately rendered images. This paper studies the gap between the gloss perceived from real-world objects and their digital counterparts. Based on our psychophysical experiments on a wide range of 3D printed samples and their corresponding photographs, we derive insights on the influence of geometry, illumination, and the display’s brightness and measure the change in gloss appearance due to the display limitations. Our evaluation experiments demonstrate that using the prediction to correct material parameters in a rendering system improves the match of gloss appearance between real objects and their visualization on a display device

The Effect of Geometry and Illumination on Appearance Perception of Different Material Categories

The understanding of material appearance perception is a complex problem due to interactions between material reflectance, surface geometry, and illumination. Recently, Serrano et al. collected the largest dataset to date with subjective ratings of material appearance attributes, including glossiness, metallicness, sharpness and contrast of reflections. In this work, we make use of their dataset to investigate for the first time the impact of the interactions between illumination, geometry, and eight different material categories in perceived appearance attributes. After an initial analysis, we select for further analysis the four material categories that cover the largest range for all perceptual attributes: fabric, plastic, ceramic, and metal. Using a cumulative link mixed model (CLMM) for robust regression, we discover interactions between these material categories and four representative illuminations and object geometries. We believe that our findings contribute to expanding the knowledge on material appearance perception and can be useful for many applications, such as scene design, where any particular material in a given shape can be aligned with dominant classes of illumination, so that a desired strength of appearance attributes can be achieved

Physiological responses of orchids to prolonged clinorotation

Creation of plant-based bioregenerative life support systems is crucial for future long-duration space exploring missions. Microgravity is one of the major stresses affecting plant growth and development under space flight conditions. Search for higher plant genotypes resilient to microgravity as well as revealing of biological features which could be used as markers of such resilience is rather urgently needed. The objective of this study was to analyze physiological and biochemical responses of three orchid species representing different life forms (terrestrial and epiphytic), growth types (monopodial and sympodial) and pathways of CO2 fixation to long-term (24 months) clinorotation which modeled the combined effect of two stress factors: hermetic conditions and microgravity. Three years old meristematic orchids Cypripedium flavum, Angraecum eburneum, Epidendrum radicans, representing different life forms, types of branching shoot system and pathways of CO2 fixation, were used as test-plants. The microgravity was simulated using three-dimensional (3-D) clinostat equipped with two rotation axes placed at right angles (rotation frequency was 3 rpm) in controlled conditions of air temperature, illumination, air humidity and substrate moisture. The&nbsp;control plants were grown in the similar plastic vessels but not hermetically sealed and without clinorotating in the same environmental conditions. The vital state of the test plants was assessed using characteristics of mineral nutrition, content of photosynthetic pigments, free amino acids, soluble proteins, DNA and RNA, enzymatic and non-enzymatic antioxidants. The&nbsp;results of this study confirmed that orchids grown under simulated microgravity and kept in hermetically-sealed vessels were subjected to oxidative stress, which could be responsible for the observed inhibition of basic physiological processes such as mineral nutrition, metabolism of aminoacids, protein biosynthesis and photosynthesis. Monopodial orchids C. flavum and A.&nbsp;eburneum demonstrated better adaptation to prolonged clinorotation as compared to sympodial E. radicans. In particular, the latter demonstrated some stimulation of mineral nutrition processes (i.e. K, N, Fe, Mn, Zn accumulation), content of photosynthetic pigments, proline and superoxide dismutase activity. Long-lasting clinorotation induced adaptive changes of antioxidant systems in the studied orchids (e.i. increase in carotenoids and proline content and stimulation of superoxide dismutase activity), which helped to maintain the main physiological functions at stable level in the above-mentioned stressful conditions. The following biochemical characteristics in the studied orchids could be considered as markers of resilience to simulated microgravity and hermetic conditions: 1) an increase in the accumulation of non-enzymatic (proline, carotenoids) and enzymatic antioxidants (superoxide dismutase); 2) ability to maintain stable balance of mineral nutrients; 3) increase in the content of photosynthetic pigments; 4)&nbsp;increase in the content of proteinogenic amino acids and soluble proteins; 5) increase in the DNA content or RNA/DNA ratio. Our studies have also demonstrated a correlation between orchid ecomorphological characteristics such as type of branching with their adaptive responses to prolonged clinorotation. We observed no correlation between the studied life form of orchids, ecotype or the pathway of CO2 fixation and their resilience to prolonged clinorotation. This research can be a starting point for studying the relationships between ecomorphological features of various orchids and their resilience to microgravity conditions in the search for biological markers of microgravity tolerance in species of higher plants

Assessing the diagnostic value of zonulin as a biomarker for intestinal permeability in patients with metabolic-associated fatty liver disease in combination with type 2 diabetes mellitus

The aim of the study was to assess the diagnostic value of serum zonulin concentrations in patients with metabolic-associated fatty liver disease (МAFLD) in combination with type 2 diabetes mellitus (T2DM). Materials and methods. The study involved 93 patients with MAFLD in combination with T2DM, who were examined and allocated to two groups. Group 1 consisted of 48 patients with non-alcoholic steatohepatitis (NASH) in combination with T2DM without small intestinal bacterial overgrowth (SIBO) syndrome. Group 2 comprised 45 patients with NASH in combination with T2DM and SIBO. The control group consisted of 25 apparently healthy persons. The ELISA method was used for quantitative determination of serum zonulin. Results. When comparing parameters of liver functional activity and ultrasonographic findings of liver steatosis and fibrosis, a significant increase in the activity of ALT and AST was revealed in Group 1 – 67.22 ± 2.25 U/l and 52.97 ± 1.04 U/l (p < 0.001) and in Group 2 – 69.20 ± 1.52 U/l and 54.82 ± 1.10 U/l (p < 0.001) compared to those in the control group – 18.00 ± 1.01 U/l and 18.96 ± 0.82 U/l (p < 0.001) respectively, as well as an increase in the ultrasound attenuation coefficient (UAC) in patients of Groups 1 and 2 amounting to 2.94 ± 0.03 dB/cm and 2.92 ± 0.04 dB/cm, respectively, and also the liver stiffness (LS) in Group 1 – 8.06 ± 0.07 kPa and in Group 2 – 8.00 ± 0.06 kPa compared to those in the control group (p < 0.001). When measuring the level of serum zonulin, a significant increase was revealed in patients of Group 1 – 61.69 ± 1.04 ng/ml and Group 2 – 89.39 ± 1.30 ng/ml compared to that in the control group – 16.76 ± 1.47 ng/ml (p < 0.001). Analyzing correlation coefficients in patients of Groups 1 and 2, a positive linear moderate association was found between the serum zonulin concentration and the activity of ALT, AST and UAC and LS. Conclusions. The study resultsobtained have demonstrated the great diagnostic value of serum zonulin as a biomarker of intestinal permeability in NASH patients in combination with T2DM, and with or without SIBO

Dataset and metrics for predicting local visible differences

A large number of imaging and computer graphics applications require localized information on the visibility of image distortions. Existing image quality metrics are not suitable for this task as they provide a single quality value per image. Existing visibility metrics produce visual difference maps, and are specifically designed for detecting just noticeable distortions but their predictions are often inaccurate. In this work, we argue that the key reason for this problem is the lack of large image collections with a good coverage of possible distortions that occur in different applications. To address the problem, we collect an extensive dataset of reference and distorted image pairs together with user markings indicating whether distortions are visible or not. We propose a statistical model that is designed for the meaningful interpretation of such data, which is affected by visual search and imprecision of manual marking. We use our dataset for training existing metrics and we demonstrate that their performance significantly improves. We show that our dataset with the proposed statistical model can be used to train a new CNN-based metric, which outperforms the existing solutions. We demonstrate the utility of such a metric in visually lossless JPEG compression, super-resolution and watermarking.</jats:p