Overcoming the limitations of commodity augmented reality head mounted displays for use in product assembly

Numerous studies have shown the effectiveness of utilizing Augmented Reality (AR) to deliver work instructions for complex assemblies. Traditionally, this research has been performed using hand-held displays, such as smartphones and tablets, or custom-built Head Mounted Displays (HMDs). AR HMDs have been shown to be especially effective for assembly tasks as they allow the user to remain hands-free while receiving work instructions. Furthermore, in recent years a wave of commodity AR HMDs have come to market including the Microsoft HoloLens, Magic Leap One, Meta 2, and DAQRI Smart Glasses. These devices present a unique opportunity for delivering assembly instructions due to their relatively low cost and accessibility compared to custom built AR HMD solutions of the past. Despite these benefits, the technology behind these HMDs still contains many limitations including input, user interface, spatial registration, navigation and occlusion. To accurately deliver work instructions for complex assemblies, the hardware limitations of these commodity AR HMDs must be overcome. For this research, an AR assembly application was developed for the Microsoft HoloLens using methods specifically designed to address the aforementioned issues. Input and user interface methods were implemented and analyzed to maximize the usability of the application. An intuitive navigation system was developed to guide users through a large training environment, leading them to the current point of interest. The native tracking system of the HoloLens was augmented with image target tracking capabilities to stabilize virtual content, enhance accuracy, and account for spatial drift. This fusion of marker-based and marker-less tracking techniques provides a novel approach to display robust AR assembly instructions on a commodity AR HMD. Furthermore, utilizing this novel spatial registration approach, the position of real-world objects was accurately registered to properly occlude virtual work instructions. To render the desired effect, specialized computer graphics methods and custom shaders were developed and implemented for an AR assembly application. After developing novel methods to display work instructions on a commodity AR HMD, it was necessary to validate that these work instructions were being accurately delivered. Utilizing the sensors on the HoloLens, data was collected during the assembly process regarding head position, orientation, assembly step times, and an estimation of spatial drift. With the addition of wearable physiological sensor data, this data was fused together in a visualization application to validate instructions were properly delivered and provide an opportunity for an analysist to examine trends within an assembly session. Additionally, the spatial drift data was then analyzed to gain a better understanding of how spatial drift accumulates over time and ensure that the spatial registration mitigation techniques was effective. Academic research has shown that AR may substantial reduce cost for assembly operations through a reduction in errors, time, and cognitive workload. This research provides novel solutions to overcome the limitations of commodity AR HMDs and validate their use for product assembly. Furthermore, the research provided in this thesis demonstrates the potential of commodity AR HMDs and how their limitations can be mitigated for use in product assembly tasks

Hardware design optimization for human motion tracking systems

A key component of any interactive computer graphics application is the system for tracking user or input device motion. An accurate estimate of the position and/or orientation of the virtual world tracking targets is critical to effectively creating a convincing virtual experience. Tracking is one of the pillars upon which a virtual reality environment is built and it imposes a fundamental limit on how real the reality of Virtual Reality can be. Whether working on a new or modified tracking system, designers typically begin the design process with requirements for the working volume, the expected user motion, and the infrastructure. Considering these requirements they develop a candidate design that includes one or more tracking mediums (optical, acoustic, etc.), associated source/sensor devices (hardware), and an algorithm (software) for combining the information from the devices. They then simulate the candidate system to estimate the performance for some specific motion paths. Thus the predictions of such traditional simulations typically include the combined effect of hardware and algorithm choices, but only for the chosen motion paths. Before tracker algorithm selection, and irrespective of the motion paths, it is the choice and configuration of the source/sensor devices that are critical to performance. The global limitations imposed by these hardware design choices set a limit on the quantity and quality of the available information (signal) for a given system configuration, and they do so in complex and sometimes unexpected ways. This complexity often makes it difficult for designers to predict or develop intuition about the expected performance impact of adding, removing, or moving source/sensor devices, changing the device parameters, etc. This research introduces a stochastic framework for evaluating and comparing the expected performance of sensing systems for interactive computer graphics. Incorporating models of the sensor devices and expected user motion dynamics, this framework enables complimentary system- and measurement-level hardware information optimization, independent of algorithm and motion paths. The approach for system-level optimization is to estimate the asymptotic position and/or orientation uncertainty at many points throughout a desired working volume or surface, and to visualize the results graphically. This global performance estimation can provide both a quantitative assessment of the expected performance and intuition about how to improve the type and arrangement of sources and sensors, in the context of the desired working volume and expected scene dynamics. Using the same model components required for these system-level optimization, the optimal sensor sampling time can be determined with respect to the expected scene dynamics for measurement-level optimization. Also presented is an experimental evaluation to support the verification of asymptotic analysis of tracking system hardware design along with theoretical analysis aimed at supporting the validity of both the system- and measurement-level optimization methods. In addition, a case study in which both the system- and measurement-level optimization methods to a working tracking system is presented. Finally, Artemis, a software tool for amplifying human intuition and experience in tracking hardware design is introduced. Artemis implements the system-level optimization framework with a visualization component for insight into hardware design choices. Like fluid flow dynamics, Artemis examines and visualizes the information flow of the source and sensor devices in a tracking system, affording interaction with the modeled devices and the resulting performance uncertainty estimate

Markerbasierte Erstellungswerkzeuge für komponentenbasierte Mixed Reality-Applikationen

In den Anwendungsbereichen der Mixed Reality (MR) werden die reale und die virtuelle Welt kombiniert, so dass ein Eindruck der Koexistenz beider Welten entsteht. Meist wird dabei die reale Umgebung durch virtuelle Objekte angereichert, die dem Anwender zusätzliche Informationen bieten sollen. Um die virtuellen Objekte richtig zu positionieren, muss die reale Umgebung erkannt werden. Diese Erkennung der realen Umgebung wird meist durch Bestimmung und Verfolgung von Orientierung und Positionierung der realen Objekte realisiert, was als Tracking bezeichnet wird und einen der wichtigsten Bestandteile für MR-Anwendung darstellt. Ohne die exakte Ausrichtung von realen und virtuellen Objekten, geht die Illusion verloren, dass die virtuellen Objekte Teil der realen Umgebung sind und mit ihr verschmelzen. Markerkombination Das markerbasierte Tracking ist ein Verfahren, das die Bestimmung der Positionierung von realen Objekten durch zusätzliche Markierungen in der realen Umgebung ermöglicht. Diese Markierungen können besonders gut durch Bildanalyseverfahren extrahiert werden und bieten anhand ihrer speziellen Form Positionierungsinformationen. Der Einsatz dieser Trackingtechnologie ist dabei denkbar einfache und kostengünstig. Ein breiter Anwendungsbereich ist durch den kostengünstigen Einsatz dieser Technologien gegeben, allerdings ist das Erstellen von MR-Anwendungen fast ausschließlich MR-Spezialisten vorbehalten, die über Programmierfertigkeiten und spezielle Kenntnisse aus dem MR-Bereich besitzen. Diese Arbeit beschreibt die Entwicklung und Umsetzung der Konzepte, die einem Personenkreis, der lediglich über geringe Kenntnisse von MR-Technologien und deren Anwendung verfügt, den kostengünstigen und einfachen Einsatz von markerbasierten Trackingtechnologien ermöglicht. Die im Rahmen der Arbeit durchgeführte Analyse verweist auf die problematischen Anwendungsfälle des markerbasierten Trackings, die durch die Verdeckung von Markern zustande kommen, in der Beschränkung der Markeranzahl begründet sind, oder durch die Schwankung der Trackingangaben entstehen. Diese Problembereiche sind bei der Entwicklung berücksichtigt worden und können mit Hilfe der entwickelten Konzepte vom Autor bewältigt werden. Das Konzept der Markerkategorien ermöglicht dabei den Einsatz von angepassten Filterungstechniken. Die redundante Markerkombination behebt das Verdeckungsproblem und eliminiert Schwankungen durch das Kombinieren von mehreren Trackinginformationen. Die Gütefunktion ermöglicht die Bewertung von Trackinginformationen und wird zur Gewichtung der Trackingangaben innerhalb einer Markerkombination genutzt. Das Konzept der Markertupel ermöglicht eine Wiederverwendung von Markern, durch den Ansatz der Bereichsunterteilung. Die Konzepte sind in der AMIRE-Umgebung vollständig implementiert und getestet worden. Zum Abschluss ist rückblickend eine kritische Betrachtung der Arbeit, in punkto Vorgehensweise und erreichter Ergebnisse durchgeführt worden.In Mixed Reality (MR) applications the real and the virtual world are combined, so that an impression of the coexistence of both worlds occurs. For the most part of MR-technology, the real environment is enriched by virtual objects, which offer additional information to the user. The real environment must be recognized to correctly position the virtual objects. The real environment is usually recognised by identifying and tracking the orientation and position of real objects. Tracking is one of the most important components for MR-applications. If the virtual objects are not accurately aligned, the illusion of the coexistence is lost. Markercombination With the marker-based tracking the positioning of the real objects is recognized by additionally attached markings. These markings can be extracted especially well by image analysis procedures. The use of this technology is economically conceivable and simple. A broad range of applications is possible by the economical use of these technologies. However developing MR-applications requires MR-specialists who have programming ability and special knowledge of MR-technology. This work describes the development and implementation of the concepts, which makes it possible for a MR-layman to independently use the marker-based tracking technology. A number of issues in marker-based tracking are covered in the report. Covering or markers, restriction in the number of markers and the fluctuation in tracking values are some of these issues. The marker categorie concept makes use of adapted filtering techniques. The redundant marker combination eliminates the covering problem and also eliminates fluctuations by combining several tracking information. The accuracy function makes the evaluation of the tracking information possible and it is used for the weighting of the tracking values in the redundant marker combination. The marker tuple concept allows re-use of markers possible, by the use of range partitioning. These concepts are fully implemented and tested in the AMIRE-authoring environment. Finally, the work and the proceeding are critically reviewed, and the results of the experiments verified

Predicting Accuracy In Pose Estimation For Marker-Based Tracking

Tracking is a necessity for interactive virtual environments. Marker-based tracking solutions involve the placement of fiducials in a rigid configuration on the object(s) to be tracked, called a tracking probe. The realization that tracking performance is linked to probe performance necessitates investigation into the design of tracking probes for proponents of marker-based tracking. A challenge involved with probe design is predicting the accuracy of a tracking probe. We present a method for predicting the accuracy of a tracking probe based upon a first-order propagation of the errors associated with the markers on the probe. Results for two sample tracking probes show excellent agreement between measured and predicted errors

Validation and uncertainty of inverse dynamics analysis applied to high acceleration movements.

This thesis is motivated by the lack of knowledge of the uncertainty in the estimation of joint forces and moments derived through inverse dynamics analysis. Previous studies have shown uncertainty bounds can be substantial during slow, simple movements such as gait or lifting however little is known about the uncertainty in inverse dynamics solutions applied to high acceleration, open chain, complex tasks. A three dimensional full body model was used to provide a mechanical basis for evaluating joint forces and moments during the golf swing. Eight male skilled golfers were used; kinematic data was recorded using the Polhemus LIBERTY, an electromagnetic tracker system, using 12 sensors attached to the body with a specially designed jacket. Force plates were used to measure ground reaction forces.Validation of the derived joint forces and moments is problematic since no 'gold standard' is available for comparison. A comparison of the measured with the estimated ground reaction forces, as well as a comparison of the moments at the T8/T9 intervertebral joint that results from bottom up and top down mechanical analysis provided an initial measure of validity. The high acceleration, complex nature of the golf swing resulted in a reduced validity compared to previous studies concerned with lifting, fast trunk rotations and slow speed golf swings. The residuals between the measured and predicted GRF were greatest during the downswing. Similarly, the residuals between the joint reaction forces and moments at the upper trunk joint measured using a top down and bottom up mechanical analysis were greatest during the downswing, exemplified by an increase in joint moment RMS differences of 30.9 Nm, 24.4 Nm and 25.2 Nm for lateral bending, axial rotation and flexion-extension respectively. It was shown that for open chain movements, through periods of high acceleration, inverse dynamics solutions can be subject to errors which have the capacity to significantly affect the interpretation of resultant joint moments depending on whether a top down or bottom up mechanical analysis is used. Top down-bottom up comparisons do not account for two sources of error; the joint centre location and the anatomical coordinate system of the joint where the two models meet. A further drawback associated with these validation methods is that nothing can be learnt about the individual sources of error and how they contribute to the total residual error.A consideration of how errors in measured variables propagate through inverse dynamics equations to produce uncertainties associated with the result was necessary. To analyse this, the Taylor Series Method for error propagation was used. Inaccuracies in body segment parameters, kinematics and external force measurement were determined experimentally. Soft tissue artefact and joint centre location errors were extracted from the literature. Inaccuracies in variables were assumed to be random and uncorrelated and results were representative of the upper bound uncertainty. Uncertainty in joint moment estimations was greatest for downswing where segments were moving with the greatest acceleration. The magnitude of the uncertainty was substantial and ranged from 6-339% of the peak joint moment magnitude.Inaccuracies in proximal moment arms and centre of mass accelerations had the most influence on the joint moment uncertainty and this uncertainty had the capability to alter the timing of peak joint moments by as much as 560ms. The results were critical to the interpretation of inverse dynamics derived joint forces and moments for high acceleration, open chain motions