Portallax:bringing 3D displays capabilities to handhelds

We present Portallax, a clip-on technology to retrofit mobile devices with 3D display capabilities. Available technologies (e.g. Nintendo 3DS or LG Optimus) and clip-on solutions (e.g. 3DeeSlide and Grilli3D) force users to have a fixed head and device positions. This is contradictory to the nature of a mobile scenario, and limits the usage of interaction techniques such as tilting the device to control a game. Portallax uses an actuated parallax barrier and face tracking to realign the barrier's position to the user's position. This allows us to provide stereo, motion parallax and perspective correction cues in 60 degrees in front of the device. Our optimized design of the barrier minimizes colour distortion, maximizes resolution and produces bigger view-zones, which support ~81% of adults' interpupillary distances and allow eye tracking implemented with the front camera. We present a reference implementation, evaluate its key features and provide example applications illustrating the potential of Portallax

Perceptually Optimized Visualization on Autostereoscopic 3D Displays

The family of displays, which aims to visualize a 3D scene with realistic depth, are known as "3D displays". Due to technical limitations and design decisions, such displays create visible distortions, which are interpreted by the human vision as artefacts. In absence of visual reference (e.g. the original scene is not available for comparison) one can improve the perceived quality of the representations by making the distortions less visible. This thesis proposes a number of signal processing techniques for decreasing the visibility of artefacts on 3D displays. The visual perception of depth is discussed, and the properties (depth cues) of a scene which the brain uses for assessing an image in 3D are identified. Following the physiology of vision, a taxonomy of 3D artefacts is proposed. The taxonomy classifies the artefacts based on their origin and on the way they are interpreted by the human visual system. The principles of operation of the most popular types of 3D displays are explained. Based on the display operation principles, 3D displays are modelled as a signal processing channel. The model is used to explain the process of introducing distortions. It also allows one to identify which optical properties of a display are most relevant to the creation of artefacts. A set of optical properties for dual-view and multiview 3D displays are identified, and a methodology for measuring them is introduced. The measurement methodology allows one to derive the angular visibility and crosstalk of each display element without the need for precision measurement equipment. Based on the measurements, a methodology for creating a quality profile of 3D displays is proposed. The quality profile can be either simulated using the angular brightness function or directly measured from a series of photographs. A comparative study introducing the measurement results on the visual quality and position of the sweet-spots of eleven 3D displays of different types is presented. Knowing the sweet-spot position and the quality profile allows for easy comparison between 3D displays. The shape and size of the passband allows depth and textures of a 3D content to be optimized for a given 3D display. Based on knowledge of 3D artefact visibility and an understanding of distortions introduced by 3D displays, a number of signal processing techniques for artefact mitigation are created. A methodology for creating anti-aliasing filters for 3D displays is proposed. For multiview displays, the methodology is extended towards so-called passband optimization which addresses Moiré, fixed-pattern-noise and ghosting artefacts, which are characteristic for such displays. Additionally, design of tuneable anti-aliasing filters is presented, along with a framework which allows the user to select the so-called 3d sharpness parameter according to his or her preferences. Finally, a set of real-time algorithms for view-point-based optimization are presented. These algorithms require active user-tracking, which is implemented as a combination of face and eye-tracking. Once the observer position is known, the image on a stereoscopic display is optimised for the derived observation angle and distance. For multiview displays, the combination of precise light re-direction and less-precise face-tracking is used for extending the head parallax. For some user-tracking algorithms, implementation details are given, regarding execution of the algorithm on a mobile device or on desktop computer with graphical accelerator

Demystifying the Future of the Screen

Demystifying the Future of the Screen explores the creation of a 3D representation of volumetric display (a graphical display device that produces 3D objects in mid-air), a technology that doesn’t yet exist in the consumer realm, using current technologies. It investigates the conceptual possibilities and technical challenges of prototyping a future, speculative, technology with current available materials. Cultural precedents, technical antecedents, economic challenges, and industry adaptation, all contribute to this thesis proposal. It pedals back to the past to examine the probable widespread integration of this future technology. By employing a detailed horizon scan, analyzing science fiction theories, and extensive user testing, I fabricated a prototype that simulates an immersive volumetric display experience, using a holographic display fan. Its construct was inspired by pre-television optical media like phantasmagoria, Kristian Birkeland’s immersive cathode-ray environments, and NBC’s original news broadcast in the early 1900s. The treatment was influenced by sci-fi film visualizations

A Review and Selective Analysis of 3D Display Technologies for Anatomical Education

The study of anatomy is complex and difficult for students in both graduate and undergraduate education. Researchers have attempted to improve anatomical education with the inclusion of three-dimensional visualization, with the prevailing finding that 3D is beneficial to students. However, there is limited research on the relative efficacy of different 3D modalities, including monoscopic, stereoscopic, and autostereoscopic displays. This study analyzes educational performance, confidence, cognitive load, visual-spatial ability, and technology acceptance in participants using autostereoscopic 3D visualization (holograms), monoscopic 3D visualization (3DPDFs), and a control visualization (2D printed images). Participants were randomized into three treatment groups: holograms (n=60), 3DPDFs (n=60), and printed images (n=59). Participants completed a pre-test followed by a self-study period using the treatment visualization. Immediately following the study period, participants completed the NASA TLX cognitive load instrument, a technology acceptance instrument, visual-spatial ability instruments, a confidence instrument, and a post-test. Post-test results showed the hologram treatment group (Mdn=80.0) performed significantly better than both 3DPDF (Mdn=66.7, p=.008) and printed images (Mdn=66.7, p=.007). Participants in the hologram and 3DPDF treatment groups reported lower cognitive load compared to the printed image treatment (p \u3c .01). Participants also responded more positively towards the holograms than printed images (p \u3c .001). Overall, the holograms demonstrated significant learning improvement over printed images and monoscopic 3DPDF models. This finding suggests additional depth cues from holographic visualization, notably head-motion parallax and stereopsis, provide substantial benefit towards understanding spatial anatomy. The reduction in cognitive load suggests monoscopic and autostereoscopic 3D may utilize the visual system more efficiently than printed images, thereby reducing mental effort during the learning process. Finally, participants reported positive perceptions of holograms suggesting implementation of holographic displays would be met with enthusiasm from student populations. These findings highlight the need for additional studies regarding the effect of novel 3D technologies on learning performance

Stereoscopic 3D user interfaces : exploring the potentials and risks of 3D displays in cars

During recent years, rapid advancements in stereoscopic digital display technology has led to acceptance of high-quality 3D in the entertainment sector and even created enthusiasm towards the technology. The advent of autostereoscopic displays (i.e., glasses-free 3D) allows for introducing 3D technology into other application domains, including but not limited to mobile devices, public displays, and automotive user interfaces - the latter of which is at the focus of this work. Prior research demonstrates that 3D improves the visualization of complex structures and augments virtual environments. We envision its use to enhance the in-car user interface by structuring the presented information via depth. Thus, content that requires attention can be shown close to the user and distances, for example to other traffic participants, gain a direct mapping in 3D space

Extending mobile touchscreen interaction

Touchscreens have become a de facto interface for mobile devices, and are penetrating further beyond their core application domain of smartphones. This work presents a design space for extending touchscreen interaction, to which new solutions may be mapped. Specific touchscreen enhancements in the domains of manual input, visual output and haptic feedback are explored and quantitative and experiental findings reported. Particular areas covered are unintentional interaction, screen locking, stereoscopic displays and picoprojection. In addition, the novel interaction approaches of finger identification and onscreen physical guides are also explored. The use of touchscreens in the domains of car dashboards and smart handbags are evaluated as domain specific use cases. This work draws together solutions from the broad area of mobile touchscreen interaction. Fruitful directions for future research are identified, and information is provided for future researchers addressing those topics.Kosketusnäytöistä on muodostunut mobiililaitteiden pääasiallinen käyttöliittymä, ja ne ovat levinneet alkuperäiseltä ydinsovellusalueeltaan, matkapuhelimista, myös muihin laitteisiin. Työssä tutkitaan uusia vuorovaikutuksen, visualisoinnin ja käyttöliittymäpalautteen keinoja, jotka laajentavat perinteistä kosketusnäytön avulla tapahtuvaa vuorovaikutusta. Näihin liittyen väitöskirjassa esitetään sekä kvantitatiivisia tuloksia että uutta kartoittavia löydöksiä. Erityisesti työ tarkastelee tahatonta kosketusnäytön käyttöä, kosketusnäytön lukitusta, stereoskooppisia kosketusnäyttöjä ja pikoprojektoreiden hyödyntämistä. Lisäksi kartoitetaan uusia vuorovaikutustapoja, jotka liittyvät sormien identifioimiseen vuorovaikutuksen yhteydessä, ja fyysisiin, liikettä ohjaaviin rakenteisiin kosketusnäytöllä. Kosketusnäytön käyttöä autossa sekä osana älykästä käsilaukkua tarkastellaan esimerkkeinä käyttökonteksteista. Väitöskirjassa esitetään vuorovaikutussuunnittelun viitekehys, joka laajentaa kosketusnäyttöjen kautta tapahtuvaa vuorovaikutusta mobiililaitteen kanssa, ja johon työssä esitellyt, uudet vuorovaikutustavat voidaan sijoittaa. Väitöskirja yhdistää kosketusnäyttöihin liittyviä käyttöliittymäsuunnittelun ratkaisuja laajalta alueelta. Työ esittelee potentiaalisia suuntaviivoja tulevaisuuden tutkimuksille ja tuo uutta tutkimustietoa, jota mobiililaitteiden vuorovaikutuksen tutkijat ja käyttöliittymäsuunnittelijat voivat hyödyntää

Extending mobile touchscreen interaction

Touchscreens have become a de facto interface for mobile devices, and are penetrating further beyond their core application domain of smartphones. This work presents a design space for extending touchscreen interaction, to which new solutions may be mapped. Specific touchscreen enhancements in the domains of manual input, visual output and haptic feedback are explored and quantitative and experiental findings reported. Particular areas covered are unintentional interaction, screen locking, stereoscopic displays and picoprojection. In addition, the novel interaction approaches of finger identification and onscreen physical guides are also explored. The use of touchscreens in the domains of car dashboards and smart handbags are evaluated as domain specific use cases. This work draws together solutions from the broad area of mobile touchscreen interaction. Fruitful directions for future research are identified, and information is provided for future researchers addressing those topics.Kosketusnäytöistä on muodostunut mobiililaitteiden pääasiallinen käyttöliittymä, ja ne ovat levinneet alkuperäiseltä ydinsovellusalueeltaan, matkapuhelimista, myös muihin laitteisiin. Työssä tutkitaan uusia vuorovaikutuksen, visualisoinnin ja käyttöliittymäpalautteen keinoja, jotka laajentavat perinteistä kosketusnäytön avulla tapahtuvaa vuorovaikutusta. Näihin liittyen väitöskirjassa esitetään sekä kvantitatiivisia tuloksia että uutta kartoittavia löydöksiä. Erityisesti työ tarkastelee tahatonta kosketusnäytön käyttöä, kosketusnäytön lukitusta, stereoskooppisia kosketusnäyttöjä ja pikoprojektoreiden hyödyntämistä. Lisäksi kartoitetaan uusia vuorovaikutustapoja, jotka liittyvät sormien identifioimiseen vuorovaikutuksen yhteydessä, ja fyysisiin, liikettä ohjaaviin rakenteisiin kosketusnäytöllä. Kosketusnäytön käyttöä autossa sekä osana älykästä käsilaukkua tarkastellaan esimerkkeinä käyttökonteksteista. Väitöskirjassa esitetään vuorovaikutussuunnittelun viitekehys, joka laajentaa kosketusnäyttöjen kautta tapahtuvaa vuorovaikutusta mobiililaitteen kanssa, ja johon työssä esitellyt, uudet vuorovaikutustavat voidaan sijoittaa. Väitöskirja yhdistää kosketusnäyttöihin liittyviä käyttöliittymäsuunnittelun ratkaisuja laajalta alueelta. Työ esittelee potentiaalisia suuntaviivoja tulevaisuuden tutkimuksille ja tuo uutta tutkimustietoa, jota mobiililaitteiden vuorovaikutuksen tutkijat ja käyttöliittymäsuunnittelijat voivat hyödyntää

Rendering and display for multi-viewer tele-immersion

Video teleconferencing systems are widely deployed for business, education and personal use to enable face-to-face communication between people at distant sites. Unfortunately, the two-dimensional video of conventional systems does not correctly convey several important non-verbal communication cues such as eye contact and gaze awareness. Tele-immersion refers to technologies aimed at providing distant users with a more compelling sense of remote presence than conventional video teleconferencing. This dissertation is concerned with the particular challenges of interaction between groups of users at remote sites. The problems of video teleconferencing are exacerbated when groups of people communicate. Ideally, a group tele-immersion system would display views of the remote site at the right size and location, from the correct viewpoint for each local user. However, is is not practical to put a camera in every possible eye location, and it is not clear how to provide each viewer with correct and unique imagery. I introduce rendering techniques and multi-view display designs to support eye contact and gaze awareness between groups of viewers at two distant sites. With a shared 2D display, virtual camera views can improve local spatial cues while preserving scene continuity, by rendering the scene from novel viewpoints that may not correspond to a physical camera. I describe several techniques, including a compact light field, a plane sweeping algorithm, a depth dependent camera model, and video-quality proxies, suitable for producing useful views of a remote scene for a group local viewers. The first novel display provides simultaneous, unique monoscopic views to several users, with fewer user position restrictions than existing autostereoscopic displays. The second is a random hole barrier autostereoscopic display that eliminates the viewing zones and user position requirements of conventional autostereoscopic displays, and provides unique 3D views for multiple users in arbitrary locations

The Turning, Stretching and Boxing Technique: a Direction Worth Looking Towards

3D avatar user interfaces (UI) are now used for many applications, a growing area for their use is serving location sensitive information to users as they need it while visiting or touring a building. Users communicate directly with an avatar rendered to a display in order to ask a question, get directions or partake in a guided tour and as a result of this kind of interaction with avatar UI, they have become a familiar part of modern human-computer interaction (HCI). However, if the viewer is not in the sweet spot (defined by Raskar et al. (1999) as a stationary viewing position at the optimal 90° angle to a 2D display) of the 2D display, the 3D illusion of the avatar deteriorates, which becomes evident as the user’s ability to interpret the avatar’s gaze direction towards points of interests (PoI) in the user’s real-world surroundings deteriorates also. This thesis combats the above problem by allowing the user to view the 3D avatar UI from outside the sweet spot, without any deterioration in the 3D illusion. The user does not lose their ability to interpret the avatar’s gaze direction and thus, the user experiences no loss in the perceived corporeal presence (Holz et al., 2011) for the avatar. This is facilitated by a three pronged graphical process called the Turning, Stretching and Boxing (TSB) technique, which maintains the avatar’s 3D illusion regardless of the user’s viewing angle and is achieved by using head-tracking data from the user captured by a Microsoft Kinect. The TSB technique is a contribution of this thesis because of how it is used with an avatar UI, where the user is free to move outside of the sweet spot without losing the 3D illusion of the rendered avatar. Then each consecutive empirical study evaluates the claims of the TSB Technique are also contributions of this thesis, those claims are as follows: (1) increase interpretability of the avatar’s gaze direction and (2) increase perception of corporeal presence for the avatar. The last of the empirical studies evaluates the use of 3D display technology in conjunction with the TSB technique. The results of Study 1 and Study 2 indicate that there is a significant increase in the participants’ abilities to interpret the avatar’s gaze direction when the TSB technique is switched on. The survey from Study 1 shows a significant increase in the perceived corporeal presence of the avatar when the TSB technique is switched on. The results from Study 3 indicate that there is no significant benefit for participants’ when interpreting the avatar’s gaze direction with 3D display technology turned on or off when the TSB technique is switched on

Quality of Experience in Immersive Video Technologies

Over the last decades, several technological revolutions have impacted the television industry, such as the shifts from black & white to color and from standard to high-definition. Nevertheless, further considerable improvements can still be achieved to provide a better multimedia experience, for example with ultra-high-definition, high dynamic range & wide color gamut, or 3D. These so-called immersive technologies aim at providing better, more realistic, and emotionally stronger experiences. To measure quality of experience (QoE), subjective evaluation is the ultimate means since it relies on a pool of human subjects. However, reliable and meaningful results can only be obtained if experiments are properly designed and conducted following a strict methodology. In this thesis, we build a rigorous framework for subjective evaluation of new types of image and video content. We propose different procedures and analysis tools for measuring QoE in immersive technologies. As immersive technologies capture more information than conventional technologies, they have the ability to provide more details, enhanced depth perception, as well as better color, contrast, and brightness. To measure the impact of immersive technologies on the viewersâ QoE, we apply the proposed framework for designing experiments and analyzing collected subjectsâ ratings. We also analyze eye movements to study human visual attention during immersive content playback. Since immersive content carries more information than conventional content, efficient compression algorithms are needed for storage and transmission using existing infrastructures. To determine the required bandwidth for high-quality transmission of immersive content, we use the proposed framework to conduct meticulous evaluations of recent image and video codecs in the context of immersive technologies. Subjective evaluation is time consuming, expensive, and is not always feasible. Consequently, researchers have developed objective metrics to automatically predict quality. To measure the performance of objective metrics in assessing immersive content quality, we perform several in-depth benchmarks of state-of-the-art and commonly used objective metrics. For this aim, we use ground truth quality scores, which are collected under our subjective evaluation framework. To improve QoE, we propose different systems for stereoscopic and autostereoscopic 3D displays in particular. The proposed systems can help reducing the artifacts generated at the visualization stage, which impact picture quality, depth quality, and visual comfort. To demonstrate the effectiveness of these systems, we use the proposed framework to measure viewersâ preference between these systems and standard 2D & 3D modes. In summary, this thesis tackles the problems of measuring, predicting, and improving QoE in immersive technologies. To address these problems, we build a rigorous framework and we apply it through several in-depth investigations. We put essential concepts of multimedia QoE under this framework. These concepts not only are of fundamental nature, but also have shown their impact in very practical applications. In particular, the JPEG, MPEG, and VCEG standardization bodies have adopted these concepts to select technologies that were proposed for standardization and to validate the resulting standards in terms of compression efficiency