Disparity-compensated view synthesis for s3D content correction

International audienceThe production of stereoscopic 3D HD content is considerably increasing and experience in 2-view acquisition is in progress. High quality material to the audience is required but not always ensured, and correction of the stereo views may be required. This is done via disparity-compensated view synthesis. A robust method has been developed dealing with these acquisition problems that introduce discomfort (e.g hyperdivergence and hyperconvergence...) as well as those ones that may disrupt the correction itself (vertical disparity, color difference between views...). The method has three phases: a preprocessing in order to correct the stereo images and estimate features (e.g. disparity range...) over the sequence. The second (main) phase proceeds then to disparity estimation and view synthesis. Dual disparity estimation based on robust block-matching, discontinuity-preserving filtering, consistency and occlusion handling has been developed. Accurate view synthesis is carried out through disparity compensation. Disparity assessment has been introduced in order to detect and quantify errors. A post-processing deals with these errors as a fallback mode. The paper focuses on disparity estimation and view synthesis of HD images. Quality assessment of synthesized views on a large set of HD video data has proved the effectiveness of our method

Objective View Synthesis Quality Assessment

International audienceView synthesis brings geometric distortions which are not handled efficiently by existing image quality assessment metrics. Despite the widespread of 3-D technology and notably 3D television (3DTV) and free-viewpoints television (FTV), the field of view synthesis quality assessment has not yet been widely investigated and new quality metrics are required. In this study, we propose a new full-reference objective quality assessment metric: the View Synthesis Quality Assessment (VSQA) metric. Our method is dedicated to artifacts detection in synthesized view-points and aims to handle areas where disparity estimation may fail: thin objects, object borders, transparency, variations of illumination or color differences between left and right views, periodic objects... The key feature of the proposed method is the use of three visibility maps which characterize complexity in terms of textures, diversity of gradient orientations and presence of high contrast. Moreover, the VSQA metric can be defined as an extension of any existing 2D image quality assessment metric. Experimental tests have shown the effectiveness of the proposed method

Multi-step flow fusion: towards accurate and dense correspondences in long video shots

International audienceThe aim of this work is to estimate dense displacement fields over long video shots. Put in sequence they are useful for representing point trajectories but also for propagating (pulling) information from a reference frame to the rest of the video. Highly elaborated optical flow estimation algorithms are at hand, and they were applied before for dense point tracking by simple accumulation, however with unavoidable position drift. On the other hand, direct long-term point matching is more robust to such deviations, but it is very sensitive to ambiguous correspondences. Why not combining the benefits of both approaches? Following this idea, we develop a multi-step flow fusion method that optimally generates dense long-term displacement fields by first merging several candidate estimated paths and then filtering the tracks in the spatio-temporal domain. Our approach permits to handle small and large displacements with improved accuracy and it is able to recover a trajectory after temporary occlusions. Especially useful for video editing applications, we attack the problem of graphic element insertion and video volume segmentation, together with a number of quantitative comparisons on ground-truth data with state-of-the-art approaches

3-D Extension and Diminishing Framework Toward Advanced Mixed Reality Systems

立命館大学博士(工学)doctoral thesi

Human factors in the perception of stereoscopic images

Research into stereoscopic displays is largely divided into how stereo 3D content looks, a field concerned with distortion, and how such content feels to the viewer, that is, comfort. However, seldom are these measures presented simultaneously. Both comfortable displays with unacceptable 3D and uncomfortable displays with great 3D are undesirable. These two scenarios can render conclusions based on research into these measures both moot and impractical. Furthermore, there is a consensus that more disparity correlates directly with greater viewer discomfort. These experiments, and the dissertation thereof, challenge this notion and argue for a more nuanced argument related to acquisition factors such as interaxial distance (IA) and post processing in the form of horizontal image translation (HIT). Indeed, this research seeks to measure tolerance limits for viewing comfort and perceptual distortions across different camera separations. In the experiments, HIT and IA were altered together. Following Banks et al. (2009), our stimuli were simple stereoscopic hinges, and we measured the perceived angle as a function of camera separation. We compared the predictions based on a ray-tracing model with the perceived 3D shape obtained psychophysically. Participants were asked to judge the angles of 250 hinges at different camera separations (IA and HIT remained linked across a 20 to 100mm range, but the angles ranged between 50° and 130°). In turn, comfort data was obtained using a five-point Likert scale for each trial. Stimuli were presented in orthoscopic conditions with screen and observer field of view (FOV) matched at 45°. The 3D hinge and experimental parameters were run across three distinct series of experiments. The first series involved replicating a typical laboratory scenario where screen position was unchanged (Experiment I), the other presenting scenarios representative of real-world applications for a single viewer (Experiments II, III, and IV), and the last presenting real-world applications for multiple viewers (Experiment V). While the laboratory scenario revealed greatest viewer comfort occurred when a virtual hinge was placed on the screen plane, the single-viewer experiment revealed into-the-screen stereo stimuli was judged flatter while out-of-screen content was perceived more veridically. The multi-viewer scenario revealed a marked decline in comfort for off-axis viewing, but no commensurate effect on distortion; importantly, hinge angles were judged as being the same regardless of off-axis viewing for angles of up to 45. More specifically, the main results are as follows. 1) Increased viewing distance enhances viewer comfort for stereoscopic perception. 2) The amount of disparity present was not correlated with comfort. Comfort is not correlated with angular distortion. 3) Distortion is affected by hinge placement on-screen. There is only a significant effect on comfort when the Camera Separation is at 60mm. 4) A perceptual bias between into the depth orientation of the screen stimuli, in to the screen stimuli were judged as flatter than out of the screen stimuli. 5) Perceived distortion not being affected by oblique viewing. Oblique viewing does not affect perceived comfort. In conclusion, the laboratory experiment highlights the limitations of extrapolating a controlled empirical stimulus into a less controlled “real world” environment. The typical usage scenarios consistently reveal no correlation between the amount of screen disparity (parallax) in the stimulus and the comfort rating. The final usage scenario reveals a perceptual constancy in off-axis viewer conditions for angles of up to 45, which, as reported, is not reflected by a typical ray-tracing model. Stereoscopic presentation with non-orthoscopic HIT may give comfortable 3D. However, there is good reason to believe that this 3D is not being perceived veridically. Comfortable 3D is often incorrectly converged due to the differences between distances specified by disparity and monocular cues. This conflict between monocular and stereo cues in the presentation of S3D content leads to loss of veridicality i.e. a perception of flatness. Therefore, correct HIT is recommended as the starting point for creating realistic and comfortable 3D, and this factor is shown by data to be far more important than limiting screen disparity (i.e. parallax). Based on these findings, this study proposes a predictive model of stereoscopic space for 3D content generators who require flexibility in acquisition parameters. This is important as there is no data for viewing conditions where the acquisition parameters are changed

Audio for Virtual, Augmented and Mixed Realities: Proceedings of ICSA 2019 ; 5th International Conference on Spatial Audio ; September 26th to 28th, 2019, Ilmenau, Germany

The ICSA 2019 focuses on a multidisciplinary bringing together of developers, scientists, users, and content creators of and for spatial audio systems and services. A special focus is on audio for so-called virtual, augmented, and mixed realities. The fields of ICSA 2019 are: - Development and scientific investigation of technical systems and services for spatial audio recording, processing and reproduction / - Creation of content for reproduction via spatial audio systems and services / - Use and application of spatial audio systems and content presentation services / - Media impact of content and spatial audio systems and services from the point of view of media science. The ICSA 2019 is organized by VDT and TU Ilmenau with support of Fraunhofer Institute for Digital Media Technology IDMT

Qualitätstaxonomie für skalierbare Algorithmen von Free Viewpoint Video Objekten

Diese Dissertation beabsichtigt einen Beitrag zur Qualitätsbeurteilung von Algorithmen für Bildanalyse und Bildsynthese im Anwendungskontext Videokommunikationssysteme zu leisten. In der vorliegenden Arbeit werden Möglichkeiten und Hindernisse der nutzerzentrierten Definition von subjektiver Qualitätswahrnehmung in diesem speziellen Anwendungsfall untersucht. Qualitätsbeurteilung von aufkommender Visualisierungs-Technologie und neuen Verfahren zur Erzeugung einer dreidimensionalen Repräsentation unter der Nutzung von Bildinformation zweier Kameras für Videokommunikationssysteme wurde bisher noch nicht umfangreich behandelt und passende Ansätze dazu fehlen. Die Herausforderungen sind es qualitätsbeeinflussende Faktoren zu definieren, passende Maße zu formulieren, sowie die Qualitätsevaluierung mit den Erstellungsalgorithmen, welche noch in Entwicklung sind, zu verbinden. Der Vorteil der Verlinkung von Qualitätswahrnehmung und Servicequalität ist die Unterstützung der technischen Realisierungsprozesse hinsichtlich ihrer Anpassungsfähigkeit (z.B. an das vom Nutzer verwendete System) und Skalierbarkeit (z.B. Beachtung eines Aufwands- oder Ressourcenlimits) unter Berücksichtigung des Endnutzers und dessen Qualitätsanforderungen. Die vorliegende Arbeit beschreibt den theoretischen Hintergrund und einen Vorschlag für eine Qualitätstaxonomie als verlinkendes Modell. Diese Arbeit beinhaltet eine Beschreibung des Projektes Skalalgo3d, welches den Rahmen der Anwendung darstellt. Präsentierte Ergebnisse bestehen aus einer systematischen Definition von qualitätsbeeinflussenden Faktoren inklusive eines Forschungsrahmens und Evaluierungsaktivitäten die mehr als 350 Testteilnehmer inkludieren, sowie daraus heraus definierte Qualitätsmerkmale der evaluierten Qualität der visuellen Repräsentation für Videokommunikationsanwendungen. Ein darauf basierendes Modell um diese Ergebnisse mit den technischen Erstellungsschritten zu verlinken wird zum Schluss anhand eines formalisierten Qualitätsmaßes präsentiert. Ein Flussdiagramm und ein Richtungsfeld zur grafischen Annäherung an eine differenzierbare Funktion möglicher Zusammenhänge werden daraufhin für weitere Untersuchungen vorgeschlagen.The thesis intends to make a contribution to the quality assessment of free viewpoint video objects within the context of video communication systems. The current work analyzes opportunities and obstacles, focusing on users' subjective quality of experience in this special case. Quality estimation of emerging free viewpoint video object technology in video communication has not yet been assessed and adequate approaches are missing. The challenges are to define factors that influence quality, to formulate an adequate measure of quality, and to link the quality of experience to the technical realization within an undefined and ever-changing technical realization process. There are two advantages of interlinking the quality of experience with the quality of service: First, it can benefit the technical realization process, in order to allow adaptability (e.g., based on systems used by the end users). Second, it provides an opportunity to support scalability in a user-centered way, e.g., based on a cost or resources limitation. The thesis outlines the theoretical background and introduces a user-centered quality taxonomy in the form of an interlinking model. A description of the related project Skalalgo3d is included, which offered a framework for application. The outlined results consist of a systematic definition of factors that influence quality, including a research framework, and evaluation activities involving more than 350 participants. The thesis includes the presentation of quality features, defined by evaluations of free viewpoint video object quality, for video communication application. Based on these quality features, a model that links these results with the technical creation process, including a formalized quality measure, is presented. Based on this, a flow chart and slope field are proposed. These intend the visualization of these potential relationships and may work as a starting point for further investigations thereon and to differentiate relations in form of functions

Curved Displays, Empirical Horopters, and Ergonomic Design Guidelines

Department of Human Factors EngineeringVisual display products should be comprehensively evaluated from the perspectives of productivity, safety, and well-being. Curved display products are known to provide advantages. Although previous studies found that curved displays increase visual task performance, reduce visual fatigue, and improve the watching experience, these studies did not comprehensively examine the effects of display curvature. Moreover, they used low-fidelity curved screens that may not effectively reflect actual curved displays. The purpose of this thesis was to develop ergonomic design guidelines for determining appropriate display curvatures, considering the productivity, safety, and well-being of visual display terminal (VDT) users. Two studies on monitors and one study on TVs were conducted for this goal. In Study 1, the effects of the display curvature, display zone, and task duration on visual task performance and visual fatigue during a visual search task on a 50-inch multi-monitor were investigated. In Study 2, the effects of the display curvature and task duration on visual task performance, visual fatigue, and user satisfaction during a proofreading task on a 27-inch monitor were investigated, and the associations between ergonomic evaluation elements were then examined. Prediction models of visual fatigue and user satisfaction were subsequently developed. In Study 3, the effects of the display curvature, viewing distance, and lateral viewing position on presence, visual comfort, and user satisfaction during a TV watching task on a 55-inch TV were examined, and the importance of six viewing experience elements affecting user satisfaction was revealed. Finally, ergonomic design guidelines for curved displays were suggested. Based on the results of studies 1 and 2, an appropriate rest-break time was ecommended, taking into account visual task performance and visual fatigue. Study 1 examined the effects of the display curvature (400 R, 600 R, 1200 R, and flat), display zone (five zones), and task duration (15 and 30 min) on legibility and visual fatigue. A total of 27 participants completed two sets of 15-minute visual search tasks with each curvature setting. The 600 R and 1200 R settings yielded better results compared to the flat setup regarding legibility and perceived visual fatigue. Relative to the corresponding center zone, the outermost zones of the 1200 R and flat settings showed a decrease of 8%???37% in legibility, whereas those of the flat environment showed an increase of 26%???45% in perceived visual fatigue. Across curvatures, legibility decreased by 2%???8%, whereas perceived visual fatigue increased by 22% during the second task set. The two task sets showed an increase of 102% in the eye complaint score and a decrease of 0.3 Hz in the critical fusion frequency, both of which indicated a rise in visual fatigue. To sum up, a curvature of around 600 R, central display zones, and frequent breaks were recommended to improve legibility and reduce visual fatigue. Study 2 examined the effects of the display curvature and task duration on proofreading performance, visual discomfort, visual fatigue, mental workload, and user satisfaction. Fifty individuals completed four 15-min proofreading tasks at a particular curvature setting. Five display curvatures (600 R, 1140 R, 2000 R, 4000 R and flat) and five task durations (0, 15, 30, 45, and 60 min) were incorporated. The mean proofreading speed at its highest when the display curvature radius was equal to the viewing distance (600 R). Across curvatures, speedaccuracy tradeoffs occurred with proofreading, as indicated by an increase of 15.5% in its mean speed and a decrease of 22.3% in its mean accuracy over one hour. Meanwhile, the mean perceived visual discomfort, subjective visual fatigue, and mental workload increased, by 54%, 74%, and 24% respectively, during the first 15-min of proofreading. A decrease of 0.4 Hz in the mean critical fusion frequency during the first 15 min and a reduction in the mean blink frequency also indicated increases in visual fatigue and mental workload. The mean user satisfaction decreased by 11% until 45 min. A segmented regression model, in which perceived visual discomfort was used as a predictor, attributed 51% of the variability to visual fatigue. To sum up, a curvature of 600 R was recommended for speedy proofreading. Moreover, the breakpoint was observed be flexible, depending on VDT task types. These findings can contribute to determining ergonomic display curvatures and scheduling interim breaks for speedy but less visually fatiguing proofreading. Study 3 examined the effects of the display curvature, viewing distance, and lateral viewing position on the TV watching experience. The watching experience was assessed regarding the spatial presence, engagement, ecological validity, negative effects, visual comfort, image quality, and display satisfaction. Four display curvatures (2.3 m, 4 m, 6 m, and flat), two viewing distances (2.3 m and 4 m), and five lateral viewing positions (0 cm, 35 cm, 70 cm, 105 cm, and 140 cm) were evaluated. Seven pairs of individuals per curvature watched ten 5 min videos together, each time at a different viewing distance and lateral viewing position. Spatial presence and engagement increased when the display curvature approached the given viewing distance. Regardless of display curvature and viewing distance and TV watching experience factors, except negative effects, were degraded at more lateral viewing positions. Engagement could effectively explain the display satisfaction. These findings can contribute to enhancing TV watching experiences by recommending specific levels of display curvatures, viewing distances, and lateral viewing positions, as well as providing information on the relative importance of each watching experience element. This work suggested ergonomic design guidelines for curved displays. In Study 1, a curvature of approximately 600 R, central display zone, and frequent breaks were proposed to improve legibility and reduce visual fatigue during visual search tasks at the viewing distance of 500 mm. In Study 2, a curvature radius of 600 R and a minimum 15-minute break interval were proposed for a speedy proofreading task, at the viewing distance of 600 mm. In Study 3, a display radius of curvature similar to the viewing distance was recommended to improve the viewing experience. These results support that a curved display is ergonomically more beneficial when the display curvature approaches the empirical horopter. A relatively short 15-minute rest-time interval was suggested, considering the decrease of task accuracy and the increase of visual fatigue in studies 1 and 2. Two regression models were selected in Study 2 regarding predictive accuracy. They accounted for 70.4% of subjective visual fatigue variability and 60.2% of user satisfaction variability. Although this work was performed using relatively higher-fidelity mock-ups than previous studies, it is necessary to verify the findings with actual curved display products in the future. Furthermore, various tasks (e.g., word processing, graphics design, and gaming) and personal characteristics (e.g., presbyopia, gender, visual acuity, and product experience) should be considered to generalize the results of this thesis. These results can contribute to determining the ergonomic display curvature in consideration of productivity, safety, and well-being, and prioritizing elements of the visual fatigue and user satisfaction resulting from VDT work.ope

Analysis of m6A mRNA modifications and their role in dosage compensation

m6A mRNA modifications play a crucial role in mRNA metabolism, in- cluding the degradation of m6A-mRNA via the YTHDF2 reader protein. To specifically understand which mRNAs are degraded in an m6A-dependent way, it is essential to locate the modification in a high-confidence and transcriptome-wide manner. m6A is commonly detected with antibody- based methods. These approaches have been shown to suffer several lim- itations due to insufficient antibody selectivities. To investigate the distri- bution of m6A, we focused in the first part of this work on improving the accurate detection of the modification. We specifically overcame limitations of m6A detection using the miCLIP protocol and established an improved miCLIP2 protocol. This was specifically coupled with an extensive bioinfor- matic pipeline and a machine learning classifier, allowing the accurate detec- tion of m6A transcriptome-wide in a single-nucleotide resolution. Using this novel high-confidence annotation, we present in the second part of the work a novel role of m6A modifications in X-to-autosome dosage compensation. X-to-autosome dosage compensation aims to balance the gene expression of autosomes and the X chromosome: While the X chromosome is present in one copy, autosomes are present in two copies. It has been proposed that X-chromosomal genes are upregulated and reach similar expression lev- els as autosomal genes. It has been shown that X-chromosomal transcripts have higher half-lives than autosomal transcritps, however, how this is ac- complished is not known. Here, we show that X-chromosomal transcripts are significantly depleted of m6A and thereby more stable than autosomal transcripts. When depleting m6A, autosomal transcripts become more stable and reach similar stabilities as the X-chromosomal transcripts. Collectively, our work firstly provides an enhanced tool for the detection of m6A in a transcriptome-wide manner and, secondly, gives novel insights into a global function of the modification in the regulation of gene expression.Getrennte Zählungen ; Illustrationen, Diagramm