3D sparse feature model using short baseline stereo and multiple view registration

This paper outlines a methodology to generate a distinctive object representation offline, using short-baseline stereo fundamentals to triangulate highly descriptive object features in multiple pairs of stereo images. A group of sparse 2.5D perspective views are built and the multiple views are then fused into a single sparse 3D model using a common 3D shape registration technique. Having prior knowledge, such as the proposed sparse feature model, is useful when detecting an object and estimating its pose for real-time systems like augmented reality

Adaptive Vision Based Scene Registration for Outdoor Augmented Reality

Augmented Reality (AR) involves adding virtual content into real scenes. Scenes are viewed using a Head-Mounted Display or other display type. In order to place content into the user's view of a scene, the user's position and orientation relative to the scene, commonly referred to as their pose, must be determined accurately. This allows the objects to be placed in the correct positions and to remain there when the user moves or the scene changes. It is achieved by tracking the user in relation to their environment using a variety of technology. One technology which has proven to provide accurate results is computer vision. Computer vision involves a computer analysing images and achieving an understanding of them. This may be locating objects such as faces in the images, or in the case of AR, determining the pose of the user. One of the ultimate goals of AR systems is to be capable of operating under any condition. For example, a computer vision system must be robust under a range of different scene types, and under unpredictable environmental conditions due to variable illumination and weather. The majority of existing literature tests algorithms under the assumption of ideal or 'normal' imaging conditions. To ensure robustness under as many circumstances as possible it is also important to evaluate the systems under adverse conditions. This thesis seeks to analyse the effects that variable illumination has on computer vision algorithms. To enable this analysis, test data is required to isolate weather and illumination effects, without other factors such as changes in viewpoint that would bias the results. A new dataset is presented which also allows controlled viewpoint differences in the presence of weather and illumination changes. This is achieved by capturing video from a camera undergoing a repeatable motion sequence. Ground truth data is stored per frame allowing images from the same position under differing environmental conditions, to be easily extracted from the videos. An in depth analysis of six detection algorithms and five matching techniques demonstrates the impact that non-uniform illumination changes can have on vision algorithms. Specifically, shadows can degrade performance and reduce confidence in the system, decrease reliability, or even completely prevent successful operation. An investigation into approaches to improve performance yields techniques that can help reduce the impact of shadows. A novel algorithm is presented that merges reference data captured at different times, resulting in reference data with minimal shadow effects. This can significantly improve performance and reliability when operating on images containing shadow effects. These advances improve the robustness of computer vision systems and extend the range of conditions in which they can operate. This can increase the usefulness of the algorithms and the AR systems that employ them

Camera pose estimation in unknown environments using a sequence of wide-baseline monocular images

In this paper, a feature-based technique for the camera pose estimation in a sequence of wide-baseline images has been proposed. Camera pose estimation is an important issue in many computer vision and robotics applications, such as, augmented reality and visual SLAM. The proposed method can track captured images taken by hand-held camera in room-sized workspaces with maximum scene depth of 3-4 meters. The system can be used in unknown environments with no additional information available from the outside world except in the first two images that are used for initialization. Pose estimation is performed using only natural feature points extracted and matched in successive images. In wide-baseline images unlike consecutive frames of a video stream, displacement of the feature points in consecutive images is notable and hence cannot be traced easily using patch-based methods. To handle this problem, a hybrid strategy is employed to obtain accurate feature correspondences. In this strategy, first initial feature correspondences are found using similarity of their descriptors and then outlier matchings are removed by applying RANSAC algorithm. Further, to provide a set of required feature matchings a mechanism based on sidelong result of robust estimator was employed. The proposed method is applied on indoor real data with images in VGA quality (640×480 pixels) and on average the translation error of camera pose is less than 2 cm which indicates the effectiveness and accuracy of the proposed approach

Scalable and Extensible Augmented Reality with Applications in Civil Infrastructure Systems.

In Civil Infrastructure System (CIS) applications, the requirement of blending synthetic and physical objects distinguishes Augmented Reality (AR) from other visualization technologies in three aspects: 1) it reinforces the connections between people and objects, and promotes engineers’ appreciation about their working context; 2) It allows engineers to perform field tasks with the awareness of both the physical and synthetic environment; 3) It offsets the significant cost of 3D Model Engineering by including the real world background. The research has successfully overcome several long-standing technical obstacles in AR and investigated technical approaches to address fundamental challenges that prevent the technology from being usefully deployed in CIS applications, such as the alignment of virtual objects with the real environment continuously across time and space; blending of virtual entities with their real background faithfully to create a sustained illusion of co- existence; integrating these methods to a scalable and extensible computing AR framework that is openly accessible to the teaching and research community, and can be readily reused and extended by other researchers and engineers. The research findings have been evaluated in several challenging CIS applications where the potential of having a significant economic and social impact is high. Examples of validation test beds implemented include an AR visual excavator-utility collision avoidance system that enables spotters to ”see” buried utilities hidden under the ground surface, thus helping prevent accidental utility strikes; an AR post-disaster reconnaissance framework that enables building inspectors to rapidly evaluate and quantify structural damage sustained by buildings in seismic events such as earthquakes or blasts; and a tabletop collaborative AR visualization framework that allows multiple users to observe and interact with visual simulations of engineering processes.PHDCivil EngineeringUniversity of Michigan, Horace H. Rackham School of Graduate Studieshttp://deepblue.lib.umich.edu/bitstream/2027.42/96145/1/dsuyang_1.pd

Tracking for Mobile 3D Augmented Reality Applications

Ph.DDOCTOR OF PHILOSOPH

Augmented Reality Framework and Demonstrator

Augmenting the real-world with digital information can improve the human perception in many ways. In recent years, a large amount of research has been conducted in the field of Augmented Reality (AR) and related technologies. Subsequently, different AR systems have been developed for the use in different areas such as medical, education, military, and entertainment. This thesis investigates augmented reality systems and challenges of realistic rendering in AR environment. Besides, an object-oriented framework, named ThirdEye, has been designed and implemented in order to facilitate the process of developing augmented reality applications for experimental purposes. This framework has been developed in two versions for desktop and mobile platforms. With ThirdEye, it is easier to port the same AR demo application to both platforms, manage and modify all AR demo application components, compared to the various existing libraries. Each feature that the ThirdEye framework includes, may be provided by other existing libraries separately but this framework provides those features in an easy-to-use manner. In order to evaluate usability and performance of ThirdEye and also for demonstrating challenges of simulating some of the light effects in the AR environment, such as shadow and refraction, several AR demos were developed using this framework. Performance of the implemented AR demos were benchmarked and bottlenecks of different components of the framework were investigated. This thesis explains the structure of the ThirdEye framework, its main components and the employed technologies and the Software Development Kits (SDKs). Furthermore, by using a simple demo, it is explained how this framework can be utilized to develop an AR application step by step. Lastly, several ideas for future development are described

Maritime Augmented Reality mit a prioriWissen aus Seekarten

The main objective of this thesis is to provide a concept to augment mar- itime sea chart information into the camera view of the user. The benefit is the simpler navigation due to the offered 3D information and the overlay onto the real 3D environment. In the maritime context special conditions hold. The sensor technologies have to be reliable in the environment of a ship’s ferrous construction. The aug- mentation of the objects has to be very precise due to the far distances of observable objects on the sea surface. Furthermore, the approach has to be reliable due to the wide range of light conditions. For a practical solution, the system has to be mobile, light-weight and with a real-time performance. To achieve this goal, the requirements are set, the possible measurement units and the data base structure are presented. First, the requirements are analyzed and a suitable system is designed. By the combination of proper sensor techniques, the local position and orienta- tion of the user can be estimated. To verify the concept, several prototypes with exchangeable units have been evaluated. This first concept is based on a marker-based approach which leads to some drawbacks. To overcome the drawbacks, the second aspect is the improvement of the sys- tem and the analysis of markerless approaches. One possible strategy will be presented. The approach uses the statistical technique of Bayesian networks to vote for single objects in the environment. By this procedure it will be shown, that due to the a priori information the underlying sea chart system has the most benefit. The analysis of the markerless approach shows, that the sea charts structure has to be adapted to the new requirements of interactive 3D augmentation scenes. After the analysis of the chart data concept, an approach for the optimization of the charts by building up an object-to-object topology within the charts data and the Bayesian object detection approach is presented. Finally, several evaluations show the performance of the imple- mented evaluation application.Diese Arbeit stellt ein Konzept zur Verfügung, um Seekarteninformationen in eine Kamera so einzublenden, dass die Informationen lagerichtig im Sichtfeld des Benutzers erscheinen. Der Mehrwert ist eine einfachere Navigation durch die Nutzung von 3D-Symbolen in der realen Umgebung. Im maritimen Umfeld gelten besondere Anforderungen an die Aufgabenstellung. Die genutzten Sensoren müssen in der Lage sein, robuste Daten in Anwesenheit der eisenhaltigen Materialien auf dem Schiff zu liefern. Die Augmentierung muss hoch genau berechnet werden, da die beobachtbaren Objekte zum Teil sehr weit entfernt auf der Meeresoberfläche verteilt sind. Weiterhin gelten die Bedingungen einer Außenumgebung, wie variierende Wetter- und Lichtbedingungen. Um eine praktikable Anwendung gewährleisten zu können, ist ein mobiles, leicht-gewichtiges und echtzeitfähiges System zu entwickeln. In dieser Arbeit werden die Anforderungen gesetzt und Konzepte für die Hardware- und Softwarelösungen beschrieben. Im ersten Teil werden die Anforderungen analysiert und ein geeignetes Hardwaresystem entwickelt. Durch die passende Kombination von Sensortechnologien kann damit die lokale Position und Orientierung des Benutzers berechnet werden. Um das Konzept zu evaluieren sind verschiedene modulare Hardware- und Softwarekonzepte als Prototypen umgesetzt worden. Das erste Softwarekonzept befasst sich mit einem markerbasierten Erkennungsalgorithmus, der in der Evaluation einige Nachteile zeigt. Dementsprechende Verbesserungen wurden in einem zweiten Softwarekonzept durch einen markerlosen Ansatz umgesetzt. Dieser Lösungsansatz nutzt Bayes'sche Netzwerke zur Erkennung einzelner Objekte in der Umgebung. Damit kann gezeigt werden, dass mit der Hilfe von a priori Informationen die dem System zugrunde liegenden Seekarten sehr gut zu diesem Zweck genutzt werden können. Die Analyse des Systemkonzeptes zeigt des weiteren, dass die Datenstruktur der Seekarten für die Anforderungen einer interaktiven, benutzergeführten 3D- Augmentierungsszene angepasst werden müssen. Nach der ausführlichen Analyse des Seekarten-Datenkonzeptes wird ein Lösungsansatz zur Optimierung der internen Seekartenstruktur aufgezeigt. Dies wird mit der Erstellung einer Objekt-zu-Objekt-Topologie in der Datenstruktur und der Verbindung zum Bayes'schen Objekterkennungsalgorithmus umgesetzt. Anschließend zeigen Evaluationen die Fähigkeiten des endgültigen Systems

Dense real-time 3D reconstruction from multiple images

The rapid increase in computer graphics and acquisition technologies has led to the widespread use of 3D models. Techniques for 3D reconstruction from multiple views aim to recover the structure of a scene and the position and orientation (motion) of the camera using only the geometrical constraints in 2D images. This problem, known as Structure from Motion (SfM) has been the focus of a great deal of research effort in recent years; however, the automatic, dense, real-time and accurate reconstruction of a scene is still a major research challenge. This thesis presents work that targets the development of efficient algorithms to produce high quality and accurate reconstructions, introducing new computer vision techniques for camera motion calibration, dense SfM reconstruction and dense real-time 3D reconstruction. In SfM, a second challenge is to build an effective reconstruction framework that provides dense and high quality surface modelling. This thesis develops a complete, automatic and flexible system with a simple user-interface of `raw images to 3D surface representation'. As part of the proposed image reconstruction approach, this thesis introduces an accurate and reliable region-growing algorithm to propagate the dense matching points from the sparse key points among all stereo pairs. This dense 3D reconstruction proposal addresses the deficiencies of existing SfM systems built on sparsely distributed 3D point clouds which are insufficient for reconstructing a complete 3D model of a scene. The existing SfM reconstruction methods perform a bundle adjustment optimization of the global geometry in order to obtain an accurate model. Such an optimization is very computational expensive and cannot be implemented in a real-time application. Extended Kalman Filter (EKF) Simultaneous Localization and Mapping (SLAM) considers the problem of concurrently estimating in real-time the structure of the surrounding world, perceived by moving sensors (cameras), simultaneously localizing in it. However, standard EKF-SLAM techniques are susceptible to errors introduced during the state prediction and measurement prediction linearization.

On object recognition for industrial augmented reality

Some reasons are market pressure, an increase of functionality, and adaptability to an already complex environment, among others. Therefore, workers face fast-changing and challenging tasks along with all the product lifecycle that reach the human cognitive limits. Although nowadays some operations are automated, many of them still need to be carried out by humans because of their complexity. In addition to management strategies and design for X, Industrial Augmented Reality (IAR) has proven to potentially benefit activities such as maintenance, assembly, manufacturing, and repair, among others. It is also supposed to upgrade the manufacturing processes by improving it, simplifying decision-making activities, reducing time and user movements, diminishing errors, and decreasing mental and physical effort. Nevertheless, IAR has not succeeded in breaking out of the laboratories and establishing itself as a strong solution in the industry, mainly because technical and interaction components are far from ideal. Its advance is limited by its enabling technologies. One of its biggest challenges are the methods for understanding the surroundings considering the different domain variables that affect IAR implementations. Thus, inspired by some systematical methodologies proposing that, for any problemsolving activity, it is required to define the characteristics that constrain the problem and the needs to be satisfied, a general frame of IAR was proposed through the identification of Domain Variables (DV), that are relevant characteristics of the industrial process in the previous Augmented Reality (AR) applications. These DV regard the user, parts, environment, and task that have an impact on the technical implementation and user performance and perception (Chapter 2). Subsequently, a detailed analysis of the influence of the DV on technical implementations related to the processes intended to understand the surroundings was performed. The results of this analysis suggest that the DV influence the technical process in two ways. The first one is that they define the boundaries in the characteristics of the technology, and the second one is that they cause some issues in the process of understanding the surroundings (Chapter 3). Further, an automatic method for creating synthetic datasets using solely the 3D model of the parts was proposed. It is hypothesized that the proposed variables are the main source of visual variations of an object in this context. Thus, the proposed method is derived from physically recreated light-matter interactions of this relevant variables. This method is aimed to create fully labeled datasets for training and testing surrounding understanding algorithms (Chapter 4). Finally, the proposed method is evaluated in a study case of object classification of two cases: a particular industrial case, and a general classification problem (using classes of ImageNet). Results suggest that fine-tuning models with the proposed method reach comparable performance (no statistical difference) than models trained with photos. These results validate the proposed method as a viable alternative for training surrounding understanding algorithms applied to industrial cases (Chapter 5)