Registration Using Projective Reconstruction for Augmented Reality Systems

In AR systems, registration is one of the most difficult problems currently limiting their applications. In this paper, we proposed a simple registration method using projective reconstruction. This method consists of two steps: embedding and tracking. Embedding involves specifying four points to build the world coordinate system on which a virtual object will be superimposed. In tracking, a projective reconstruction technique in computer vision is used to track the four specified points to compute the modelview transformation for augmentation. This method is simple as only four points need to be specified at the embedding stage, and the virtual object can then be easily augmented in a real video sequence. In addition, it can be extended to a common scenario using a common projective matrix. The proposed method has three advantages: (1) It is fast because the linear least square method can be used to estimate the related matrix in the algorithm and it is not necessary to calculate the fundamental matrix in the extended case; (2) A virtual object can still be superimposed on a related area even if some parts of the specified area are occluded during the augmentation process; and (3) This method is robust because it remains effective even when not all the reference points are detected during the augmentation process (in the rendering process), as long as at least six pairs of related reference point correspondences can be found. Several projective matrices obtained from the authors’ previous work, which are unrelated with the present AR system, were tested on this extended registration method. Experiments showed that these projective matrices can also be utilized for tracking the specified points.Singapore-MIT Alliance (SMA

Urbanmix

“Realidad aumentada”, se puede definir como la amplificación de la capacidad sensorial de la percepción del mundo que nos rodea mediante el uso de dispositivos que superponen elementos virtuales a la imagen real. La aplicación Urbanmix, nos posibilita este concepto a escala urbana y en exteriores usando un sistema propio de realidad aumentada, que se basa en modelos 3D y el programa Max/MSP Jitter [2] de edición de video en tiempo real. A través de la aplicación Urbanmix, podemos acceder a una visión de la ciudad o espacio en el que nos encontremos, ampliada con algunos edificios o monumentos provenientes de otras. Con ello, y utilizando escalas 1:1 en el sistema, podemos realizar fácilmente comparaciones reales y así tomar verdadera conciencia de la magnitud de los edificios que visualizamos. Ante la gran dimensión de algunos de los edificios que podemos introducir en el sistema, es necesario ubicarlos a una distancia lo suficientemente grande como para que el sistema pueda tomar las referencias necesarias para ubicarlo correctamente y su visualización continúe siendo creíble; por la falta de espacio en las ciudades lo normal es que algunos edificios reales se superpongan generalmente a los virtuales, por una superposición de canales de video, con lo que nos encontramos con un problema de oclusión de formas reales a virtuales que no se corresponde con lo habitual en estos sistemas, oclusión de formas virtuales a reales. El problema se resuelve con una mascara tridimensional de video que actualiza sus datos en tiempo real, que permite al usuario moverse a su antojo dentro del área acotada en donde se desarrolle la aplicación, y continuar viendo la inmersión del edificio o monumento escogido en el espacio precargado desde cualquier punto de vista

Entorn de desenvolupament de realitat augmentada

La Realitat Augmentada és un camp en ple auge investigador. En aquest projecte proposem un entorn amb el qual poder prototipar tant aplicacions d'usuari com algoritmes associats a aquesta tecnologia. En aquesta memòria es recullen l'estudi previ, el disseny i els detalls d'implementació de l'entorn proposat així com una solució específica de Realitat Augmentada associada a aquest entorn basada en visió per computador. Finalment, es presenten els resultats d'una anàlisi de rendiment i de disseny del projecte.La Realidad Aumentada es un campo en pleno auge investigador. En este proyecto proponemos un entorno con el cual poder prototipar tanto aplicaciones de usuario como algoritmos asociados a esta tecnología. En esta memoria se recogen el estudio previo, el diseño y los detalles de implementación del entorno propuesto así como una solución específica de Realidad Aumentada asociada a este entorno basada en visión por computador. Finalmente, se presentan los resultados de un análisis de rendimiento y de diseño del proyecto.Augmented Reality it's a booming research field. In this project we propose an environment with which both prototype user applications and algorithms associated with this technology. This report reflects the previous study, design and implementation details of the proposed environment as well as an Augmented Reality solution associated with this environment based on computer vision. Finally, we present the results of performance and project design analysis

Object Registration in Semi-cluttered and Partial-occluded Scenes for Augmented Reality

This paper proposes a stable and accurate object registration pipeline for markerless augmented reality applications. We present two novel algorithms for object recognition and matching to improve the registration accuracy from model to scene transformation via point cloud fusion. Whilst the first algorithm effectively deals with simple scenes with few object occlusions, the second algorithm handles cluttered scenes with partial occlusions for robust real-time object recognition and matching. The computational framework includes a locally supported Gaussian weight function to enable repeatable detection of 3D descriptors. We apply a bilateral filtering and outlier removal to preserve edges of point cloud and remove some interference points in order to increase matching accuracy. Extensive experiments have been carried to compare the proposed algorithms with four most used methods. Results show improved performance of the algorithms in terms of computational speed, camera tracking and object matching errors in semi-cluttered and partial-occluded scenes

Corrección del error en el proceso de registro en los sistemas de realidad aumentada utilizando técnicas heurísticas

Desde la masificación de los primeros computadores en la década de los 90’s su uso estaba restringido a desarrollar tareas específicas y cálculos repetitivos. En la actualidad, los computadores y sistemas informáticos están permeando cada vez más las actividades humanas, convirtiéndose en lo que Weiser define como Sistemas Ubicuos [Wei99]. Este nuevo paradigma de interacción humano-máquina implica disponer de interfaces naturales que permitan una comunicación efectiva entre el usuario y la máquina, presentando grandes retos para las ciencias de la computación en cuanto al diseño de sistemas, su modelamiento y, en particular, el desarrollo de las interfaces de usuario. En este contexto, las interfaces deben entrar en consonancia con el concepto de Amplificación de la Inteligencia [Bro96][Bro96a] tal que permita la creación de sistemas capaces de amplificar o mejorar las capaces cognitivas humanas en lugar de imitarlas o reemplazarlas. En esta búsqueda de interfaces cada vez más naturales que extiendan las capacidades perceptivas humanas, surge la creación de nuevos entornos y metáforas de visualización como la Realidad Virtual (RV) y la Realidad Aumentada (RA). En particular, la realidad aumentada aparece como un nuevo paradigma de visualización e interacción humano-máquina que permite al usuario obtener información adicional (información virtual) de su entorno e incluso manipular esta información conservando su relación con el ambiente real. La obtención de dicha información adicional, se logra mediante la superposición en el ambiente real, de información virtual generada por computador. La creación de un sistema de realidad aumentada involucra varias etapas, a saber: calibración de dispositivos, extracción de características, seguimiento y registro. Esta última etapa presenta grandes retos para la creación de un ambiente de RA realista, ya que aquí se une tanto la información virtual como la real; si dicha alineación no es correcta, el sistema será visualmente incoherente. Es por esto que el registro es una etapa crítica que actualmente limita las aplicaciones de realidad aumentada. En la presente tesis, se aborda por medio de técnicas heurísticas esta limitación, que ha sido recurrente en la literatura. Como aporte se propone el uso de técnicas heurísticas, las cuales hasta ahora no han sido abordadas en la literatura, para disminuir el error existente entre la información de posicionamiento obtenida en etapas anteriores al proceso de registro (información estimada) y la información real. Dicha disminución en el error se traduce en un alineamiento real-virtual (etapa de registro) mucho más preciso y coherente, obteniendo en consecuencia sistemas o ambientes que apoyen de manera efectiva los problemas de visualización y acceso a información en una mayor cantidad de aplicaciones./Abstarct. Since the first computers appear in the early 90’s, its use was restricted to perform specific tasks and repetitive calculations. Today, computers are getting mixed in human activities more and more, becoming what Weiser [Wei99] defined as ubiquitous systems. This new paradigm of human-machine interaction implies the availability of natural interfaces that allow an effective communication between the user and the machine. It presents great challenges for computer science related to the systems design and modeling, and in particular, the development of user interfaces. In this context, the interfaces must be in relation with the concept of Intelligence Amplification [Bro96][Bro96a], that allows the creation of systems that can amplify or improve human cognitive perceptions rather than imitate or replace them. In this search for more and more natural interfaces that extend human cognitive capacities, new environments and visualization metaphors such as Virtual Reality (VR) and Augmented Reality (AR) rose. In particular, augmented reality appears as a new paradigm for visualization and human-machine interaction that allows the user to obtain additional information (virtual information) from his/her environment and even manipulate this information while preserving their relationship with the real world. Obtaining such information is achieved by the overlap in the real environment, virtual information generated by computer. The creation of an augmented reality system involves several steps, namely, device calibration, feature extraction, tracking and registration. This last stage presents great challenges for creating an realistic AR environment, since this stage binds both the virtual and the real information, if the alignment is not correct, the system will be visually incoherent. This is why the registration is a critical stage which currently limits the applications of augmented reality. In this thesis, this limitation is tackle by means of heuristics. Contributions include the use of heuristics to reduce the error between the position information obtained in previous stages to the registration process (information estimated) and real data. This reduction in error resulting in a real-virtual alignment (registration stage) much more accurate and consistent, thus gaining support to systems and environments to solve effectively visualization problems and access to information in a larger number of applications.Maestrí

A STUDY OF THE APPLICATION OF AUGMENTED REALITY TECHNOLOGY ON 3-AXIS CNC IN SITU MACHINING SIMULATION

Ph.DDOCTOR OF PHILOSOPH

Dense real-time 3D reconstruction from multiple images

The rapid increase in computer graphics and acquisition technologies has led to the widespread use of 3D models. Techniques for 3D reconstruction from multiple views aim to recover the structure of a scene and the position and orientation (motion) of the camera using only the geometrical constraints in 2D images. This problem, known as Structure from Motion (SfM) has been the focus of a great deal of research effort in recent years; however, the automatic, dense, real-time and accurate reconstruction of a scene is still a major research challenge. This thesis presents work that targets the development of efficient algorithms to produce high quality and accurate reconstructions, introducing new computer vision techniques for camera motion calibration, dense SfM reconstruction and dense real-time 3D reconstruction. In SfM, a second challenge is to build an effective reconstruction framework that provides dense and high quality surface modelling. This thesis develops a complete, automatic and flexible system with a simple user-interface of `raw images to 3D surface representation'. As part of the proposed image reconstruction approach, this thesis introduces an accurate and reliable region-growing algorithm to propagate the dense matching points from the sparse key points among all stereo pairs. This dense 3D reconstruction proposal addresses the deficiencies of existing SfM systems built on sparsely distributed 3D point clouds which are insufficient for reconstructing a complete 3D model of a scene. The existing SfM reconstruction methods perform a bundle adjustment optimization of the global geometry in order to obtain an accurate model. Such an optimization is very computational expensive and cannot be implemented in a real-time application. Extended Kalman Filter (EKF) Simultaneous Localization and Mapping (SLAM) considers the problem of concurrently estimating in real-time the structure of the surrounding world, perceived by moving sensors (cameras), simultaneously localizing in it. However, standard EKF-SLAM techniques are susceptible to errors introduced during the state prediction and measurement prediction linearization.

Entornos multimedia de realidad aumentada en el campo del arte

La relación ente Ciencia y Arte ha mantenido a lo largo de la historia momentos de proximidad o distanciamiento, llegando a entenderse como dos culturas diferentes, pero también se han producido situaciones interdisciplinares de colaboración e intercambio que en nuestros días mantienen como nexo común la cultura digital y el uso del ordenador. Según Berenguer (2002) desde la aparición del ordenador, científicos y artistas están encontrando un espacio común de trabajo y entendimiento. Mediante el empleo de las nuevas tecnologías, la distancia que separa ambas disciplinas es cada vez más corta. En esta tesis, cuyo título es "Entornos Multimedia de Realidad Aumentada en el Campo del Arte", se presenta una investigación teórico-práctica de la tecnología de realidad aumentada aplicada al arte y campos afines, como el edutainment (educación + entretenimiento). La investigación se ha realizado en dos bloques: en el primer bloque se trata la tecnología desde distintos factores que se han considerado relevantes para su entendimiento y funcionamiento; en el segundo se presentan un total de seis ensayos que constituyen la parte práctica de esta tesis.Portalés Ricart, C. (2008). Entornos multimedia de realidad aumentada en el campo del arte [Tesis doctoral no publicada]. Universitat Politècnica de València. https://doi.org/10.4995/Thesis/10251/3402Palanci

Localisation and tracking of stationary users for extended reality

In this thesis, we investigate the topics of localisation and tracking in the context of Extended Reality. In many on-site or outdoor Augmented Reality (AR) applications, users are standing or sitting in one place and performing mostly rotational movements, i.e. stationary. This type of stationary motion also occurs in Virtual Reality (VR) applications such as panorama capture by moving a camera in a circle. Both applications require us to track the motion of a camera in potentially very large and open environments. State-of-the-art methods such as Structure-from-Motion (SfM), and Simultaneous Localisation and Mapping (SLAM), tend to rely on scene reconstruction from significant translational motion in order to compute camera positions. This can often lead to failure in application scenarios such as tracking for seated sport spectators, or stereo panorama capture where the translational movement is small compared to the scale of the environment. To begin with, we investigate the topic of localisation as it is key to providing global context for many stationary applications. To achieve this, we capture our own datasets in a variety of large open spaces including two sports stadia. We then develop and investigate these techniques in the context of these sports stadia using a variety of state-of-the-art localisation approaches. We cover geometry-based methods to handle dynamic aspects of a stadium environment, as well as appearance-based methods, and compare them to a state-of-the-art SfM system to identify the most applicable methods for server-based and on-device localisation. Recent work in SfM has shown that the type of stationary motion that we target can be reliably estimated by applying spherical constraints to the pose estimation. In this thesis, we extend these concepts into a real-time keyframe-based SLAM system for the purposes of AR, and develop a unique data structure for simplifying keyframe selection. We show that our constrained approach can track more robustly in these challenging stationary scenarios compared to state-of-the-art SLAM through both synthetic and real-data tests. In the application of capturing stereo panoramas for VR, this thesis demonstrates the unsuitability of standard SfM techniques for reconstructing these circular videos. We apply and extend recent research in spherically constrained SfM to creating stereo panoramas and compare this with state-of-the-art general SfM in a technical evaluation. With a user study, we show that the motion requirements of our SfM approach are similar to the natural motion of users, and that a constrained SfM approach is sufficient for providing stereoscopic effects when viewing the panoramas in VR

Advances in top-down and bottom-up approaches to video-based camera tracking

Video-based camera tracking consists in trailing the three dimensional pose followed by a mobile camera using video as sole input. In order to estimate the pose of a camera with respect to a real scene, one or more three dimensional references are needed. Examples of such references are landmarks with known geometric shape, or objects for which a model is generated beforehand. By comparing what is seen by a camera with what is geometrically known from reality, it is possible to recover the pose of the camera that is sensing these references. In this thesis, we investigate the problem of camera tracking at two levels. Firstly, we work at the low level of feature point recognition. Feature points are used as references for tracking and we propose a method to robustly recognise them. More specifically, we introduce a rotation-discriminative region descriptor and an efficient rotation-discriminative method to match feature point descriptors. The descriptor is based on orientation gradient histograms and template intensity information. Secondly, we have worked at the higher level of camera tracking and propose a fusion of top-down (TDA) and bottom-up approaches (BUA). We combine marker-based tracking using a BUA and feature points recognised from a TDA into a particle filter. Feature points are recognised with the method described before. We take advantage of the identification of the rotation of points for tracking purposes. The goal of the fusion is to take advantage of their compensated strengths. In particular, we are interested in covering the main capabilities that a camera tracker should provide. These capabilities are automatic initialisation, automatic recovery after loss of track, and tracking beyond references known a priori. Experiments have been performed at the two levels of investigation. Firstly, tests have been conducted to evaluate the performance of the recognition method proposed. The assessment consists in a set of patches extracted from eight textured images. The images are rotated and matching is done for each patch. The results show that the method is capable of matching accurately despite the rotations. A comparison with similar techniques in the state of the art depicts the equal or even higher precision of our method with much lower computational cost. Secondly, experimental assessment of the tracking system is also conducted. The evaluation consists in four sequences with specific problematic situations namely, occlusions of the marker, illumination changes, and erratic and/or fast motion. Results show that the fusion tracker solves characteristic failure modes of the two combined approaches. A comparison with similar trackers shows competitive accuracy. In addition, the three capabilities stated earlier are fulfilled in our tracker, whereas the state of the art reveals that no other published tracker covers these three capabilities simultaneously. The camera tracking system has a potential application in the robotics domain. It has been successfully used as a man-machine interface and applied in Augmented Reality environments. In particular, the system has been used by students of the University of art and design Lausanne (ECAL) with the purpose of conceiving new interaction concepts. Moreover, in collaboration with ECAL and fabric | ch (studio for architecture & research), we have jointly developed the Augmented interactive Reality Toolkit (AiRToolkit). The system has also proved to be reliable in public events and is the basis of a game-oriented demonstrator installed in the Swiss National Museum of Audiovisual and Multimedia (Audiorama) in Montreux