Towards markerless orthopaedic navigation with intuitive Optical See-through Head-mounted displays

The potential of image-guided orthopaedic navigation to improve surgical outcomes has been well-recognised during the last two decades. According to the tracked pose of target bone, the anatomical information and preoperative plans are updated and displayed to surgeons, so that they can follow the guidance to reach the goal with higher accuracy, efficiency and reproducibility. Despite their success, current orthopaedic navigation systems have two main limitations: for target tracking, artificial markers have to be drilled into the bone and calibrated manually to the bone, which introduces the risk of additional harm to patients and increases operating complexity; for guidance visualisation, surgeons have to shift their attention from the patient to an external 2D monitor, which is disruptive and can be mentally stressful. Motivated by these limitations, this thesis explores the development of an intuitive, compact and reliable navigation system for orthopaedic surgery. To this end, conventional marker-based tracking is replaced by a novel markerless tracking algorithm, and the 2D display is replaced by a 3D holographic Optical see-through (OST) Head-mounted display (HMD) precisely calibrated to a user's perspective. Our markerless tracking, facilitated by a commercial RGBD camera, is achieved through deep learning-based bone segmentation followed by real-time pose registration. For robust segmentation, a new network is designed and efficiently augmented by a synthetic dataset. Our segmentation network outperforms the state-of-the-art regarding occlusion-robustness, device-agnostic behaviour, and target generalisability. For reliable pose registration, a novel Bounded Iterative Closest Point (BICP) workflow is proposed. The improved markerless tracking can achieve a clinically acceptable error of 0.95 deg and 2.17 mm according to a phantom test. OST displays allow ubiquitous enrichment of perceived real world with contextually blended virtual aids through semi-transparent glasses. They have been recognised as a suitable visual tool for surgical assistance, since they do not hinder the surgeon's natural eyesight and require no attention shift or perspective conversion. The OST calibration is crucial to ensure locational-coherent surgical guidance. Current calibration methods are either human error-prone or hardly applicable to commercial devices. To this end, we propose an offline camera-based calibration method that is highly accurate yet easy to implement in commercial products, and an online alignment-based refinement that is user-centric and robust against user error. The proposed methods are proven to be superior to other similar State-of- the-art (SOTA)s regarding calibration convenience and display accuracy. Motivated by the ambition to develop the world's first markerless OST navigation system, we integrated the developed markerless tracking and calibration scheme into a complete navigation workflow designed for femur drilling tasks during knee replacement surgery. We verify the usability of our designed OST system with an experienced orthopaedic surgeon by a cadaver study. Our test validates the potential of the proposed markerless navigation system for surgical assistance, although further improvement is required for clinical acceptance.Open Acces

Looking at instructional animations through the frame of virtual camera

This thesis investigates the virtual camera and the function of camera movements in expository motion graphics for the purpose of instruction. Motion graphic design is a popular video production technique often employed to create instructional animations that present educational content through the persuasive presentation styles of the entertainment media industry. Adopting animation as a learning tool has distinct concerns and challenges when compared to its use in entertainment, and combining cognitive learning and emotive design aspects requires additional design considerations for each design element. The thesis will address how the camera movement-effect in supporting the narrative and aesthetic in instructional animations. It does this by investigating the virtual camera in terms of technical, semiotic and psychological level, culminating in a systematic categorization of functional camera movements on the basis of conceptual framework that describes hybrid integration of physical, cognitive and affective design aspects; and a creative work as a case study in the form of a comprehensive instructional animation that demonstrates practiced camera movements. Due to the correlation of the conceptual framework relied upon by the supplementary work with the techniques of effective instructional video production and conventional entertainment filmmaking, this thesis touches on the relationship between live action and animation in terms of directing and staging, concluding that the virtual camera as a design factor can be useful for supporting a narrative, evoking emotion and directing the audience’s focus while revealing, tracking and emphasizing informatio

Auditory Displays and Assistive Technologies: the use of head movements by visually impaired individuals and their implementation in binaural interfaces

Visually impaired people rely upon audition for a variety of purposes, among these are the use of sound to identify the position of objects in their surrounding environment. This is limited not just to localising sound emitting objects, but also obstacles and environmental boundaries, thanks to their ability to extract information from reverberation and sound reflections- all of which can contribute to effective and safe navigation, as well as serving a function in certain assistive technologies thanks to the advent of binaural auditory virtual reality. It is known that head movements in the presence of sound elicit changes in the acoustical signals which arrive at each ear, and these changes can improve common auditory localisation problems in headphone-based auditory virtual reality, such as front-to-back reversals. The goal of the work presented here is to investigate whether the visually impaired naturally engage head movement to facilitate auditory perception and to what extent it may be applicable to the design of virtual auditory assistive technology. Three novel experiments are presented; a field study of head movement behaviour during navigation, a questionnaire assessing the self-reported use of head movement in auditory perception by visually impaired individuals (each comparing visually impaired and sighted participants) and an acoustical analysis of inter-aural differences and cross- correlations as a function of head angle and sound source distance. It is found that visually impaired people self-report using head movement for auditory distance perception. This is supported by head movements observed during the field study, whilst the acoustical analysis showed that interaural correlations for sound sources within 5m of the listener were reduced as head angle or distance to sound source were increased, and that interaural differences and correlations in reflected sound were generally lower than that of direct sound. Subsequently, relevant guidelines for designers of assistive auditory virtual reality are proposed

Viewing 3D TV over two months produces no discernible effects on balance, coordination or eyesight

With the rise in stereoscopic 3D media, there has been concern that viewing stereoscopic 3D (S3D) content could have long-term adverse effects, but little data are available. In the first study to address this, 28 households who did not currently own a 3D TV were given a new TV set, either S3D or 2D. The 116 members of these households all underwent tests of balance, coordination and eyesight, both before they received their new TV set, and after they had owned it for 2 months. We did not detect any changes which appeared to be associated with viewing 3D TV. We conclude that viewing 3D TV does not produce detectable effects on balance, coordination or eyesight over the timescale studied. Practitioner Summary: Concern has been expressed over possible long-term effects of stereoscopic 3D (S3D). We looked for any changes in vision, balance and coordination associated with normal home S3D TV viewing in the 2 months after first acquiring a 3D TV. We find no evidence of any changes over this timescale

Stereoscopic bimanual interaction for 3D visualization

Virtual Environments (VE) are being widely used in various research fields for several decades such as 3D visualization, education, training and games. VEs have the potential to enhance the visualization and act as a general medium for human-computer interaction (HCI). However, limited research has evaluated virtual reality (VR) display technologies, monocular and binocular depth cues, for human depth perception of volumetric (non-polygonal) datasets. In addition, a lack of standardization of three-dimensional (3D) user interfaces (UI) makes it challenging to interact with many VE systems. To address these issues, this dissertation focuses on evaluation of effects of stereoscopic and head-coupled displays on depth judgment of volumetric dataset. It also focuses on evaluation of a two-handed view manipulation techniques which support simultaneous 7 degree-of-freedom (DOF) navigation (x,y,z + yaw,pitch,roll + scale) in a multi-scale virtual environment (MSVE). Furthermore, this dissertation evaluates auto-adjustment of stereo view parameters techniques for stereoscopic fusion problems in a MSVE. Next, this dissertation presents a bimanual, hybrid user interface which combines traditional tracking devices with computer-vision based "natural" 3D inputs for multi-dimensional visualization in a semi-immersive desktop VR system. In conclusion, this dissertation provides a guideline for research design for evaluating UI and interaction techniques

Distance Perception Through Head-Mounted Displays

It has been shown in numerous research studies that people tend to underestimate distances while wearing head-mounted displays (HMDs). We investigated various possible factors affecting the perception of distance is HMDs through multiple studies. Many contributing factors has been identified by researchers in the past decades, however, further investigation is required to provide a better understanding of this problem. In order to find a baseline for distance underestimation, we performed a study to compare the distance perception in real world versus a fake headset versus a see-through HMD. Users underestimated distances while wearing the fake headset or the see-through HMD. The fake headset and see-through HMD had similar result, while they had significant difference with the real-world results. Since the fake headset and the HMD had similar underestimation results, we decided to focus on the FOV of the headset which was a common factor between these two conditions. To understand the effects of FOV on the perception of distance in a virtual environment we performed a study through a blind-throwing task. FOVs at three different diagonal angles, 60°, 110° and 200° were compared with each other. The results showed people underestimate the distances more in restricted FOVs. As this study was performed using static 360° images of a single environment, we decided to see if the results can be extended to various 3D environments. A mixed-design study to compare the effect of horizontal FOV and vertical FOV on egocentric distance perception in four different realistic VEs was performed. The results indicated more accurate distance judgement with larger horizontal FOV with no significant effect of vertical FOV. More accurate distance judgement in indoor VEs compared to outdoor VEs was observed. Also, participants judged distances more accurately in cluttered environments versus uncluttered environments. These results highlights the importance of the environment in distance-critical VR applications and also shows that wider horizontal FOV should be considered for an improved distance judgment

Autostereoscopy vs. non-autostereoscopy on the LG Optimus 3D

[ES] El rápido aumento de los dispositivos con características 3D ha permitido una serie de sistemas de entretenimiento nuevos y avanzados para casa, hecho que ha aumentado la demanda de contenidos en 3D: películas en 3D, series en 3D y videojuegos en 3D. Esta tecnología ya se ha aplicado en las pantallas de los teléfonos inteligentes y videoconsolas portátiles. En esta tesina, se realizó un estudio sobre una aplicación para dispositivos Android con dos modos de visualización e interacción con el usuario: con autoestereoscopía y sin autoestereoscopía. Esta aplicación contiene varios objetos, tanto estáticos como dinámicos, en un entorno 3D. Tras realizar la evaluación, los resultados indican el alto grado de interés que tienen los contenidos en 3D para juegos en teléfonos inteligentes. Sin embargo, los problemas de percepción de objetos virtuales en 3D demuestran que esta tecnología todavía necesita mejoras para proporcionar una percepción de profundidad sin pérdida de nitidez en la imagen para que sea adecuada a un amplio grupo de la población.[EN] The rapid increase of 3D capable devices has provided a series of new and advanced home entertainment systems; that indicates a higher number of demands for 3D contents, such as 3D movies, 3D TV series and 3D games. As a result, this technology has been applied already on the displays of Smartphones and handheld video gaming consoles. In this thesis, a study between autostereoscopy and non-autostereoscopy on a Smartphone was carried out by testing a new Android application that provides both visualization modes with user interactions. The new app contains a number of static and dynamic objects in a 3D environment. Evaluation findings indicate that people are interested in 3D game content on Smartphones. However, perception issues of 3D virtual objects and loss of picture quality demonstrate that this technology still needs further improvements before it can become suitable for all groups of people.Kaczmarczyk, KM. (2013). Autostereoscopy vs. non-autostereoscopy on the LG Optimus 3D. http://hdl.handle.net/10251/37109Archivo delegad

Ergonomics of the Operative Field in Paediatric Minimal Access Surgery

Imperial Users onl

Immersive moodboards, a comparative study of industrial design inspiration material

A recent trend in our industrial culture has been the gradual emergence of digital tools in various fields of human activity. They aim to reduce development time, cost, or to insure a low error, high quality process. Many fields have been improved thanks to this new computerized approach. This paper is centered on how industrial design could be assisted by virtual reality tools. More specifically it presents a new vision of early design methodologies through immersive technologies. It also presents the results of an experimentation aiming to compare traditional moodboards with a newly developed immersive moodboard. When analyzing and comparing the relationship of the industrial designer to a traditional and an immersive moodboard, our result highlights the fact that immerging the industrial designer in an immersive moodboard induces a high emotional activity without radically modifying the meaning of the represented trend. The moodboard data spatialization stimulates and engages the designer into interacting with the immersive moodboard. The virtual reality system provides the illusion of a potential reality, which can be used by the designer as a reflection basis for his work. We believe that delivering this immersive experience during the early design process will help the industrial designer make style related decisions.Projet CARNO