Integrating augmented reality data into a mobile simulation framework

Augmented reality devices can be used by engineers and scientists to perform numerical simulations for different physical systems. Using augmented reality glasses for presenting simulation results, in the form of virtual objects, improves interactivity and quality of user experience (QoE). However, despite availability of efficient approaches for fast approximate computations, such as the reduced basis method (RBM), it is not feasible to preload all reduced models for different simulations and sets of parameters in advance. Therefore, an approach is presented, that allows to detect real-world objects, map specific tags to corresponding simulations and to efficiently use available storage. By analyzing the environmental sensor data and estimating the required quality, determined by user perception, only relevant reduced bases are loaded to the internal storage of mobile device. The proposed system periodically obtains solution for the given simulation in accordance with changed parameters and quality demands, either by loading updated reduced model from server or performing the computation of the full problem directly on the mobile device in case of network failure. Simulation results are visualized by means of hologram, overlaid onto the detected object with exact position and pose. Evaluation results show that solving the full numerical problem on mobile device is feasible for dimensions up to 32x32, whereas for higher quality constraints the latency of requesting the reduced model from server can be improved by up to 10 times by performing preliminary availability check and starting computation in advance

Causal inference in generalizable environments: systematic representative design

Causal inference and generalizability both matter. Historically, systematic designs emphasize causal inference, while representative designs focus on generalizability. Here, we suggest a transformative synthesis – Systematic Representative Design (SRD) – concurrently enhancing both causal inference and “built-in” generalizability by leveraging today’s intelligent agent, virtual environments, and other technologies. In SRD, a “default control group” (DCG) can be created in a virtual environment by representatively sampling from real-world situations. Experimental groups can be built with systematic manipulations onto the DCG base. Applying systematic design features (e.g., random assignment to DCG versus experimental groups) in SRD affords valid causal inferences. After explicating the proposed SRD synthesis, we delineate how the approach concurrently advances generalizability and robustness, cause-effect inference and precision science, a computationally-enabled cumulative psychological science supporting both “bigger theory” and concrete implementations grappling with tough questions (e.g., what is context?) and affording rapidly-scalable interventions for real-world problems

Towards markerless orthopaedic navigation with intuitive Optical See-through Head-mounted displays

The potential of image-guided orthopaedic navigation to improve surgical outcomes has been well-recognised during the last two decades. According to the tracked pose of target bone, the anatomical information and preoperative plans are updated and displayed to surgeons, so that they can follow the guidance to reach the goal with higher accuracy, efficiency and reproducibility. Despite their success, current orthopaedic navigation systems have two main limitations: for target tracking, artificial markers have to be drilled into the bone and calibrated manually to the bone, which introduces the risk of additional harm to patients and increases operating complexity; for guidance visualisation, surgeons have to shift their attention from the patient to an external 2D monitor, which is disruptive and can be mentally stressful. Motivated by these limitations, this thesis explores the development of an intuitive, compact and reliable navigation system for orthopaedic surgery. To this end, conventional marker-based tracking is replaced by a novel markerless tracking algorithm, and the 2D display is replaced by a 3D holographic Optical see-through (OST) Head-mounted display (HMD) precisely calibrated to a user's perspective. Our markerless tracking, facilitated by a commercial RGBD camera, is achieved through deep learning-based bone segmentation followed by real-time pose registration. For robust segmentation, a new network is designed and efficiently augmented by a synthetic dataset. Our segmentation network outperforms the state-of-the-art regarding occlusion-robustness, device-agnostic behaviour, and target generalisability. For reliable pose registration, a novel Bounded Iterative Closest Point (BICP) workflow is proposed. The improved markerless tracking can achieve a clinically acceptable error of 0.95 deg and 2.17 mm according to a phantom test. OST displays allow ubiquitous enrichment of perceived real world with contextually blended virtual aids through semi-transparent glasses. They have been recognised as a suitable visual tool for surgical assistance, since they do not hinder the surgeon's natural eyesight and require no attention shift or perspective conversion. The OST calibration is crucial to ensure locational-coherent surgical guidance. Current calibration methods are either human error-prone or hardly applicable to commercial devices. To this end, we propose an offline camera-based calibration method that is highly accurate yet easy to implement in commercial products, and an online alignment-based refinement that is user-centric and robust against user error. The proposed methods are proven to be superior to other similar State-of- the-art (SOTA)s regarding calibration convenience and display accuracy. Motivated by the ambition to develop the world's first markerless OST navigation system, we integrated the developed markerless tracking and calibration scheme into a complete navigation workflow designed for femur drilling tasks during knee replacement surgery. We verify the usability of our designed OST system with an experienced orthopaedic surgeon by a cadaver study. Our test validates the potential of the proposed markerless navigation system for surgical assistance, although further improvement is required for clinical acceptance.Open Acces

Leveraging Augmented Reality for Real-time Operational Performance Management

Augmented Reality (AR) projects a virtual overlay onto real space so that the user can see a superimposed image over the real-world background. Although AR has advanced recently and a breadth of applications can be found in practice, they are focused on simple tasks with few examples of more complex work tasks. One area that could benefit from advancing AR technology is operations management, specifically operational performance measurement (OPM); however, a brief review of the literature reveals that this potential application area has not yet been explored. Therefore, the purpose of this work is to investigate the application of AR technology to OPM to improve real-time decision-making and management practice. A systematic literature review was conducted to evaluate the current application areas related to management practices. This review did not identify any studies related to using AR to support OPM, but did identify many applications relevant to management activities that empirically demonstrate the benefit of adoption. The review analyzed the current development in this research area and how it has matured including evaluating the applications discussed in the identified publications to demonstrate the existing gap in the research related to OPM applications. An expert study was then conducted to explore potential challenges and benefits of such a device as well as to operationally define effective decision-making for operations managers. The results of the expert study were leveraged to develop a Design of Experiments based laboratory study to empirically test the effects of an AR supported environment on decision-making effectiveness and operational performance. The results showed that the AR device supported improved operational performance, but did not show a significant effect on participants\u27 perceived decision-making effectiveness. This study contributes to the academic literature on technology-enabled OPM and managerial decision-making as well as providing insights for industry professionals interested in adopting AR to support management functions

Visual access to lifelog data in a virtual environment

Continuous image capture via a wearable camera is currently one of the most popular methods to establish a comprehensive record of the entirety of an indi- vidual’s life experience, referred to in the research community as a lifelog. These vast image corpora are further enriched by content analysis and combined with additional data such as biometrics to generate as extensive a record of a person’s life as possible. However, interfacing with such datasets remains an active area of research, and despite the advent of new technology and a plethora of com- peting mediums for processing digital information, there has been little focus on newly emerging platforms such as virtual reality. We hypothesise that the increase in immersion, accessible spatial dimensions, and more, could provide significant benefits in the lifelogging domain over more conventional media. In this work, we motivate virtual reality as a viable method of lifelog exploration by performing an in-depth analysis using a novel application prototype built for the HTC Vive. This research also includes the development of a governing design framework for lifelog applications which supported the development of our prototype but is also intended to support the development of future such lifelog systems