Augmented Reality Interfaces for Enabling Fast and Accurate Task Localization

Changing viewpoints is a common technique to gain additional visual information about the spatial relations among the objects contained within an environment. In many cases, all of the necessary visual information is not available from a single vantage point, due to factors such as occlusion, level of detail, and limited field of view. In certain instances, strategic viewpoints may need to be visited multiple times (e.g., after each step of an iterative process), which makes being able to transition between viewpoints precisely and with minimum effort advantageous for improved task performance (e.g., faster completion time, fewer errors, less dependence on memory). Many augmented reality (AR) applications are designed to make tasks easier to perform by supplementing a user's first-person view with virtual instructions. For those tasks that benefit from being seen from more than a single viewpoint, AR users typically have to physically relocalize (i.e., move a see-through display and typically themselves since those displays are often head-worn or hand-held) to those additional viewpoints. However, this physical motion may be costly or difficult, due to increased distances or obstacles in the environment. We have developed a set of interaction techniques that enable fast and accurate task localization in AR. Our first technique, SnapAR, allows users to take snapshots of augmented scenes that can be virtually revisited at later times. The system stores still images of scenes along with camera poses, so that augmentations remain dynamic and interactive. Our prototype implementation features a set of interaction techniques specifically designed to enable quick viewpoint switching. A formal evaluation of the capability to manipulate virtual objects within snapshot mode showed significant savings in time spent and gain in accuracy when compared to physically traveling between viewpoints. For cases when a user has to physically travel to a strategic viewpoint (e.g., to perform maintenance and repair on a large physical piece of equipment), we present ParaFrustum, a geometric construct that represents this set of strategic viewpoints and viewing directions and establishes constraints on a range of acceptable locations for the user's eyes and a range of acceptable angles in which the user's head can be oriented. Providing tolerance in the allowable viewing positions and directions avoids burdening the user with the need to assume a tightly constrained 6DOF pose when it is not required by the task. We describe two visualization techniques, ParaFrustum-InSitu and ParaFrustum-HUD, that guide a user to assume one of the poses defined by a ParaFrustum. A formal user study corroborated that speed improvements increase with larger tolerances and reveals interesting differences in participant trajectories based on the visualization technique. When the object to be operated on is smaller and can be handheld, instead of being large and stationary, it can be manually rotated instead of the user moving to a strategic viewpoint. Examples of such situations include tasks in which one object must be oriented relative to a second prior to assembly and tasks in which objects must be held in specific ways to inspect them. Researchers have investigated guidance mechanisms for some 6DOF tasks, using wide--field-of-view (FOV), stereoscopic virtual and augmented reality head-worn displays (HWDs). However, there has been relatively little work directed toward smaller FOV lightweight monoscopic HWDs, such as Google Glass, which may remain more comfortable and less intrusive than stereoscopic HWDs in the near future. In our Orientation Assistance work, we have designed and implemented a novel visualization approach and three additional visualizations representing different paradigms for guiding unconstrained manual 3DOF rotation, targeting these monoscopic HWDs. This chapter includes our exploration of these paradigms and the results of a user study evaluating the relative performance of the visualizations and showing the advantages of our new approach. In summary, we investigated ways of enabling an AR user to obtain visual information from multiple viewpoints, both physically and virtually. In the virtual case, we showed how one can change viewpoints precisely and with less effort. In the physical case, we explored how we can interactively guide users to obtain strategic viewpoints, either by moving their heads or re-orienting handheld objects. In both cases, we showed that our techniques help users accomplish certain types of tasks more quickly and with fewer errors, compared to when they have to change viewpoints following alternative, previously suggested methods

Immersive augmented reality system for the training of pattern classification control with a myoelectric prosthesis

Background!#!Hand amputation can have a truly debilitating impact on the life of the affected person. A multifunctional myoelectric prosthesis controlled using pattern classification can be used to restore some of the lost motor abilities. However, learning to control an advanced prosthesis can be a challenging task, but virtual and augmented reality (AR) provide means to create an engaging and motivating training.!##!Methods!#!In this study, we present a novel training framework that integrates virtual elements within a real scene (AR) while allowing the view from the first-person perspective. The framework was evaluated in 13 able-bodied subjects and a limb-deficient person divided into intervention (IG) and control (CG) groups. The IG received training by performing simulated clothespin task and both groups conducted a pre- and posttest with a real prosthesis. When training with the AR, the subjects received visual feedback on the generated grasping force. The main outcome measure was the number of pins that were successfully transferred within 20 min (task duration), while the number of dropped and broken pins were also registered. The participants were asked to score the difficulty of the real task (posttest), fun-factor and motivation, as well as the utility of the feedback.!##!Results!#!The performance (median/interquartile range) consistently increased during the training sessions (4/3 to 22/4). While the results were similar for the two groups in the pretest, the performance improved in the posttest only in IG. In addition, the subjects in IG transferred significantly more pins (28/10.5 versus 14.5/11), and dropped (1/2.5 versus 3.5/2) and broke (5/3.8 versus 14.5/9) significantly fewer pins in the posttest compared to CG. The participants in IG assigned (mean ± std) significantly lower scores to the difficulty compared to CG (5.2 ± 1.9 versus 7.1 ± 0.9), and they highly rated the fun factor (8.7 ± 1.3) and usefulness of feedback (8.5 ± 1.7).!##!Conclusion!#!The results demonstrated that the proposed AR system allows for the transfer of skills from the simulated to the real task while providing a positive user experience. The present study demonstrates the effectiveness and flexibility of the proposed AR framework. Importantly, the developed system is open source and available for download and further development

ARVISCOPE: Georeferenced Visualization of Dynamic Construction Processes in Three-Dimensional Outdoor Augmented Reality.

Construction processes can be conceived as systems of discrete, interdependent activities. Discrete Event Simulation (DES) has thus evolved as an effective tool to model operations that compete over available resources (personnel, material, and equipment). A DES model has to be verified and validated to ensure that it reflects a modeler’s intentions, and faithfully represents a real operation. 3D visualization is an effective means of achieving this, and facilitating the process of communicating and accrediting simulation results. Visualization of simulated operations has traditionally been achieved in Virtual Reality (VR). In order to create convincing VR animations, detailed information about an operation and the environment has to be obtained. The data must describe the simulated processes, and provide 3D CAD models of project resources, the facility under construction, and the surrounding terrain (Model Engineering). As the size and complexity of an operation increase, such data collection becomes an arduous, impractical, and often impossible task. This directly translates into loss of financial and human resources that could otherwise be productively used. In an effort to remedy this situation, this dissertation proposes an alternate approach of visualizing simulated operations using Augmented Reality (AR) to create mixed views of real existing jobsite facilities and virtual CAD models of construction resources. The application of AR in animating simulated operations has significant potential in reducing the aforementioned Model Engineering and data collection tasks, and at the same time can help in creating visually convincing output that can be effectively communicated. This dissertation presents the design, methodology, and development of ARVISCOPE, a general purpose AR animation authoring language, and ROVER, a mobile computing hardware framework. When used together, ARVISCOPE and ROVER can create three-dimensional AR animations of any length and complexity from the results of running DES models of engineering operations. ARVISCOPE takes advantage of advanced Global Positioning System (GPS) and orientation tracking technologies to accurately track a user’s spatial context, and georeferences superimposed 3D graphics in an augmented environment. In achieving the research objectives, major technical challenges such as accurate registration, automated occlusion handling, and dynamic scene construction and manipulation have been successfully identified and addressed.Ph.D.Civil EngineeringUniversity of Michigan, Horace H. Rackham School of Graduate Studieshttp://deepblue.lib.umich.edu/bitstream/2027.42/60761/1/abehzada_1.pd

Use of live video overlay on 3D data for distributed collaborative review

Thesis (S.M.)--Massachusetts Institute of Technology, School of Architecture and Planning, Program in Media Arts and Sciences, 2013.Cataloged from PDF version of thesis.Includes bibliographical references (p. 84-85).Using everyday skills, such as pointing and drawing freehand sketches, facilitates effective communication when reviewing visual information. However, for sharing three-dimensional (3D) data, it is difficult to support such approaches of using hands and real ink in a seamless way. This thesis proposes a new system design called AnnoScape as an approach to performing a remote collaborative review of 3D digital data using a live video overlay of the desktop image on the viewports of the 3D scene. The system's virtual viewports are controlled with tangible handles and can be left spatially in the 3D data space. The viewports can be shared with remote collaborators both asynchronously and in real time. The system allows distributed users to navigate shared 3D space individually or jointly (synchronizing the viewport); generate an overlay of the live video of hand drawings, physical objects, and printed images from the desktop surface with the viewport; and control the legibility of the visual contents. This spatial video overlay technique in the 3D data space allows distributed users to share the live annotations over the synchronized viewports. We report the prototype design and initial experiments to explore AnnoScape's possibilities through the scenario of having remote collaborators review the exterior site and interior reconfiguration of an existing architectural setting.by Austin Seungmin Lee.S.M

3D Design Review Systems in Immersive Environments

Design reviews play a crucial role in the development process, ensuring the quality and effectiveness of designs in various industries. However, traditional design review methods face challenges in effectively understanding and communicating complex 3D models. Immersive technologies, particularly Head-Mounted Displays (HMDs), offer new opportunities to enhance the design review process. In this thesis, we investigate using immersive environments, specifically HMDs, for 3D design reviews. We begin with a systematic literature review to understand the current state of employing HMDs in industry for design reviews. As part of this review, we utilize a detailed taxonomy from the literature to categorize and analyze existing approaches. Additionally, we present four iterations of an immersive design review system developed during my industry experience. Two of these iterations are evaluated through case studies involving domain experts, including engineers, designers, and clients. A formal semi-structured focus group is conducted to gain further insights into traditional design review practices. The outcomes of these evaluations and the focus group discussions are thoroughly discussed. Based on the literature review and the focus group findings, we uncover a new challenge associated with using HMDs in immersive design reviews—asynchronous and remote collaboration. Unlike traditional design reviews, where participants view the same section on a shared screen, HMDs allow independent exploration of areas of interest, leading to a shift from synchronous to asynchronous communication. Consequently, important feedback may be missed as the lead designer disconnects from the users' perspectives. To address this challenge, we collaborate with a domain expert to develop a prototype that utilizes heatmap visualization to display 3D gaze data distribution. This prototype enables lead designers to quickly identify areas of review and missed regions. The study incorporates the Design Critique approach and provides valuable insights into different heatmap visualization variants (top view projection, object-based, and volume-based). Furthermore, a list of well-defined requirements is outlined for future spatio-temporal visualization applications aimed at integrating into existing workflows. Overall, this thesis contributes to the understanding and improvement of immersive design review systems, particularly in the context of utilizing HMDs. It offers insights into the current state of employing HMDs for design reviews, utilizes a taxonomy from the literature to analyze existing approaches, highlights challenges associated with asynchronous collaboration, and proposes a prototype solution with heatmap visualization to address the identified challenge

Advances in top-down and bottom-up approaches to video-based camera tracking

Video-based camera tracking consists in trailing the three dimensional pose followed by a mobile camera using video as sole input. In order to estimate the pose of a camera with respect to a real scene, one or more three dimensional references are needed. Examples of such references are landmarks with known geometric shape, or objects for which a model is generated beforehand. By comparing what is seen by a camera with what is geometrically known from reality, it is possible to recover the pose of the camera that is sensing these references. In this thesis, we investigate the problem of camera tracking at two levels. Firstly, we work at the low level of feature point recognition. Feature points are used as references for tracking and we propose a method to robustly recognise them. More specifically, we introduce a rotation-discriminative region descriptor and an efficient rotation-discriminative method to match feature point descriptors. The descriptor is based on orientation gradient histograms and template intensity information. Secondly, we have worked at the higher level of camera tracking and propose a fusion of top-down (TDA) and bottom-up approaches (BUA). We combine marker-based tracking using a BUA and feature points recognised from a TDA into a particle filter. Feature points are recognised with the method described before. We take advantage of the identification of the rotation of points for tracking purposes. The goal of the fusion is to take advantage of their compensated strengths. In particular, we are interested in covering the main capabilities that a camera tracker should provide. These capabilities are automatic initialisation, automatic recovery after loss of track, and tracking beyond references known a priori. Experiments have been performed at the two levels of investigation. Firstly, tests have been conducted to evaluate the performance of the recognition method proposed. The assessment consists in a set of patches extracted from eight textured images. The images are rotated and matching is done for each patch. The results show that the method is capable of matching accurately despite the rotations. A comparison with similar techniques in the state of the art depicts the equal or even higher precision of our method with much lower computational cost. Secondly, experimental assessment of the tracking system is also conducted. The evaluation consists in four sequences with specific problematic situations namely, occlusions of the marker, illumination changes, and erratic and/or fast motion. Results show that the fusion tracker solves characteristic failure modes of the two combined approaches. A comparison with similar trackers shows competitive accuracy. In addition, the three capabilities stated earlier are fulfilled in our tracker, whereas the state of the art reveals that no other published tracker covers these three capabilities simultaneously. The camera tracking system has a potential application in the robotics domain. It has been successfully used as a man-machine interface and applied in Augmented Reality environments. In particular, the system has been used by students of the University of art and design Lausanne (ECAL) with the purpose of conceiving new interaction concepts. Moreover, in collaboration with ECAL and fabric | ch (studio for architecture & research), we have jointly developed the Augmented interactive Reality Toolkit (AiRToolkit). The system has also proved to be reliable in public events and is the basis of a game-oriented demonstrator installed in the Swiss National Museum of Audiovisual and Multimedia (Audiorama) in Montreux

Human-Mechanical system interaction in Virtual Reality

The present work aims to show the great potential of Virtual Reality (VR) technologies in the field of Human-Robot Interaction (HRI). Indeed, it is foreseeable that in not too distant future cooperating robots will be increasingly present in human environments. Many authors actually believe that after the current information revolution, we will witness the so-called "robotics revolution", with the spread of increasingly intelligent and autonomous robots capable of moving into our own environments. Since these machines must be able to interact with human beings in a safe way, new design tools for the study of Human-Robot Interaction (HRI) are needed. The author believes that VR is an ideal design tool for the study of the interaction between humans and automatic machines, since it allows the designers to interact in real-time with virtual robotic systems and to evaluate different control algorithms, without the need of physical prototypes. This also shields the user from any risk related to the physical experimentation. However, VR technologies have also a more immediate application in the field of HRI, such as the study of usability of interfaces for real-time controlled robots. In fact, these robots, such as robots for microsurgery or even "teleoperated" robots working in a hostile environments, are already quite common. VR allows the designers to evaluate the usability of such interfaces by relating their physical input with a virtual output. In particular, the author has developed a new software application aimed at simulating automatic robots and, more generally, mechanical systems in a virtual environment. The user can interact with one or more virtual manipulators and also control them in real-time by means of several input devices. Finally, an innovative approach to the modeling and control of a humanoid robot with high degree of redundancy is discussed. VR implementation of a virtual humanoid is useful for the study of both humanoid robots and human beings

GIS-based landscape design research

Landscape design research is important for cultivating spatial intelligence in landscape architecture. This study explores GIS (geographic information systems) as a tool for landscape design research - investigating landscape designs to understand them as architectonic compositions (architectonic plan analysis). The concept ‘composition’ refers to a conceivable arrangement, an architectural expression of a mental construct that is legible and open to interpretation. Landscape architectonic compositions and their representations embody a great wealth of design knowledge as objects of our material culture and reflect the possible treatment of the ground, space, image and program as a characteristic coherence. By exploring landscape architectonic compositions with GIS, design researchers can acquire design knowledge that can be used in the creation and refinement of a design.  The research aims to identify and illustrate the potential role of GIS as a tool in landscape design research, so as to provide insight into the possibilities and limitations of using GIS in this capacity. The critical, information-oriented case of Stourhead landscape garden (Wiltshire, UK), an example of a designed landscape that covers the scope and remit of landscape architecture design, forms the heart of the study. The exploration of Stourhead by means of GIS can be understood as a plausibility probe. Here the case study is considered a form of ‘quasi-experiment’, testing the hypothesis and generating a learning process that constitutes a prerequisite for advanced understanding, while using an adjusted version of the framework for landscape design analysis by Steenbergen and Reh (2003). This is a theoretically informed analytical method based on the formal interpretation of the landscape architectonic composition addressing four landscape architectonic categories: the basic, the spatial, the symbolic and the programmatic form. This study includes new aspects to be analysed, such as the visible form and the shape of the walk, and serves as the basis for the landscape architectonic analysis in which GIS is used as the primary analytical tool.  GIS-based design research has the possibility to cultivate spatial intelligence in landscape architecture through three fields of operation: GIS-based modelling: description of existing and future landscape architectonic compositions in digital form; GIS-based analysis: exploration, analysis and synthesis of landscape architectonic compositions in order to reveal latent architectonic relationships and principles, while utilizing the processing capacities and possibilities of computers for ex-ante and ex-post simulation and evaluation; GIS-based visual representation: representation of (virtual) landscape architectonic compositions in space and time, in order to retrieve and communicate information and knowledge of the landscape design.  Though there are limitations, this study exemplifies that GIS is a powerful instrument to acquire knowledge from landscape architectonic compositions. The study points out that the application of GIS in landscape design research can be seen as an extension of the fundamental cycle of observation, visual representation, analysis and interpretation in the process of knowledge acquisition, with alternative visualisations and digital landscape models as important means for this process. Using the calculating power of computers, combined with inventive modelling, analysis and visualisation concepts in an interactive process, opened up possibilities to reveal new information and knowledge about the basic, spatial, symbolic and programmatic form of Stourhead. GIS extended the design researchers’ perception via measurement, simulation and experimentation, and at the same time offered alternative ways of understanding the landscape architectonic composition. This gave rise to the possibility of exploring new elements in the framework of landscape design research, such as the visible form and kinaesthetic aspects, analysing the composition from eyelevel perspective. Moreover, the case study showcases that GIS has the potential to measure phenomena that are often subject to intuitive and experimental design, combining general scientific knowledge of, for instance, visual perception and way-finding, with the examination of site-specific design applications. GIS also enabled one to understand the landscape architectonic composition of Stourhead as a product of time, via the analysis of its development through reconstruction and evaluation of several crucial time-slice snapshots. The study illustrates that GIS can be regarded an external cognitive tool that facilitates and mediates in design knowledge acquisition. GIS facilitates in the sense that it can address the ‘same types of design-knowledge’ regarding the basic, spatial, symbolic and programmatic form, but in a more precise, systematic, transparent, and quantified manner. GIS mediates in the sense that it influences what and how aspects of the composition can be understood and therefore enables design researchers to generate ‘new types of design-knowledge’ by advanced spatial analysis and the possibility of linking or integrating other information layers, fields of science and data sources. The research contributes to the development and distribution of knowledge of GIS-applications in landscape architecture in two ways: (1) by ‘following’ the discipline and developing aspects of it, and (2) by setting in motion fundamental developments in the field, providing alternative readings of landscape architecture designs

GIS-based landscape design research:

Landscape design research is important for cultivating spatial intelligence in landscape architecture. This study explores GIS (geographic information systems) as a tool for landscape design research - investigating landscape designs to understand them as architectonic compositions (architectonic plan analysis). The concept ‘composition’ refers to a conceivable arrangement, an architectural expression of a mental construct that is legible and open to interpretation. Landscape architectonic compositions and their representations embody a great wealth of design knowledge as objects of our material culture and reflect the possible treatment of the ground, space, image and program as a characteristic coherence. By exploring landscape architectonic compositions with GIS, design researchers can acquire design knowledge that can be used in the creation and refinement of a design.  The research aims to identify and illustrate the potential role of GIS as a tool in landscape design research, so as to provide insight into the possibilities and limitations of using GIS in this capacity. The critical, information-oriented case of Stourhead landscape garden (Wiltshire, UK), an example of a designed landscape that covers the scope and remit of landscape architecture design, forms the heart of the study. The exploration of Stourhead by means of GIS can be understood as a plausibility probe. Here the case study is considered a form of ‘quasi-experiment’, testing the hypothesis and generating a learning process that constitutes a prerequisite for advanced understanding, while using an adjusted version of the framework for landscape design analysis by Steenbergen and Reh (2003). This is a theoretically informed analytical method based on the formal interpretation of the landscape architectonic composition addressing four landscape architectonic categories: the basic, the spatial, the symbolic and the programmatic form. This study includes new aspects to be analysed, such as the visible form and the shape of the walk, and serves as the basis for the landscape architectonic analysis in which GIS is used as the primary analytical tool.  GIS-based design research has the possibility to cultivate spatial intelligence in landscape architecture through three fields of operation: GIS-based modelling: description of existing and future landscape architectonic compositions in digital form; GIS-based analysis: exploration, analysis and synthesis of landscape architectonic compositions in order to reveal latent architectonic relationships and principles, while utilizing the processing capacities and possibilities of computers for ex-ante and ex-post simulation and evaluation; GIS-based visual representation: representation of (virtual) landscape architectonic compositions in space and time, in order to retrieve and communicate information and knowledge of the landscape design.  Though there are limitations, this study exemplifies that GIS is a powerful instrument to acquire knowledge from landscape architectonic compositions. The study points out that the application of GIS in landscape design research can be seen as an extension of the fundamental cycle of observation, visual representation, analysis and interpretation in the process of knowledge acquisition, with alternative visualisations and digital landscape models as important means for this process. Using the calculating power of computers, combined with inventive modelling, analysis and visualisation concepts in an interactive process, opened up possibilities to reveal new information and knowledge about the basic, spatial, symbolic and programmatic form of Stourhead. GIS extended the design researchers’ perception via measurement, simulation and experimentation, and at the same time offered alternative ways of understanding the landscape architectonic composition. This gave rise to the possibility of exploring new elements in the framework of landscape design research, such as the visible form and kinaesthetic aspects, analysing the composition from eyelevel perspective. Moreover, the case study showcases that GIS has the potential to measure phenomena that are often subject to intuitive and experimental design, combining general scientific knowledge of, for instance, visual perception and way-finding, with the examination of site-specific design applications. GIS also enabled one to understand the landscape architectonic composition of Stourhead as a product of time, via the analysis of its development through reconstruction and evaluation of several crucial time-slice snapshots. The study illustrates that GIS can be regarded an external cognitive tool that facilitates and mediates in design knowledge acquisition. GIS facilitates in the sense that it can address the ‘same types of design-knowledge’ regarding the basic, spatial, symbolic and programmatic form, but in a more precise, systematic, transparent, and quantified manner. GIS mediates in the sense that it influences what and how aspects of the composition can be understood and therefore enables design researchers to generate ‘new types of design-knowledge’ by advanced spatial analysis and the possibility of linking or integrating other information layers, fields of science and data sources. The research contributes to the development and distribution of knowledge of GIS-applications in landscape architecture in two ways: (1) by ‘following’ the discipline and developing aspects of it, and (2) by setting in motion fundamental developments in the field, providing alternative readings of landscape architecture designs