Precise localization for aerial inspection using augmented reality markers

The final publication is available at link.springer.comThis chapter is devoted to explaining a method for precise localization using augmented reality markers. This method can achieve precision of less of 5 mm in position at a distance of 0.7 m, using a visual mark of 17 mm × 17 mm, and it can be used by controller when the aerial robot is doing a manipulation task. The localization method is based on optimizing the alignment of deformable contours from textureless images working from the raw vertexes of the observed contour. The algorithm optimizes the alignment of the XOR area computed by means of computer graphics clipping techniques. The method can run at 25 frames per second.Peer ReviewedPostprint (author's final draft

Alakzatok lineáris deformációinak becslése és orvosi alkalmazásai = Estimation of Linear Shape Deformations and its Medical Applications

A projekt fő eredménye egy általánosan használható, teljesen automatikus alakzat regisztrációs módszer, amely az alábbi tulajdonságokkal rendelkezik: • nincs szükség pontmegfeleltetésekre illetve iteratív optimalizáló algoritmusokra; • képes 2D lineáris és (invertálható) projektív deformációk, valamint 3D affin deformációk meghatározására; • robusztus a geometriai és szegmentálási hibákra; • lineáris időkomplexitású, ami lehetővé teszi nagy felbontású képek közel valós idejű illesztését. Publikusan elérhetővé tettünk 3 demó programot, amelyek a 2D és 3D affin, valamint síkhomográfia regisztrációs algoritmusainkat implementálják. Továbbá kifejlesztettünk egy prototípus szoftvert csípőprotézis röntgenképek illesztésére, amit átadtunk a projektben közreműködő radiológusoknak további felhasználásra. Az eredményeinket a terület vezető konferenciáin ( pl. ICCV, ECCV) illetve vezető folyóiratokban (pl. IEEE Transactions on Pattern Analysis and Machine Intelligence, Pattern Recognition). A projekten dolgozó egyik MSc hallgató második helyezést ért el az OTDK-n. Domokos Csaba PhD fokozatot szerzett, továbbá munkáját Kuba Attila díjjal ismerte el a Képfeldolgozók és Alakfelismerők Társasága. A projekt eredményeiről részletesebb információ a projekt honlapokon található: • http://www.inf.u-szeged.hu/ipcg/projects/AFFSHAPE.html • http://www.inf.u-szeged.hu/ipcg/projects/AffinePuzzle.html • http://www.inf.u-szeged.hu/ipcg/projects/diffeoshape.html | The main achievement of the project is a fully functional automatic shape registration method with the following properties: • it doesn’t need established point correspondences nor the use of iterative optimization algorithms; • capable of recovering 2D linear and (invertible) projective shape deformations as well as affine distortions of 3D shapes; • robust in the presence of geometric noise and segmentation errors; • has a linear time complexity allowing near real-time registration of high resolution images. 3 demo programs are publicly available implementing our affine 2D, 3D and planar homography registration algorithms. Furthermore, we have developed a prototype software for aligning hip prosthesis X-ray images, which has been transfered to collaborating radiologists for further exploitation. Our results have been presented at top conferences (e.g. ICCV, ECCV) and in leading journals (e.g. IEEE Trans. on Patt. Anal. & Mach. Intell., Patt. Rec.). An MSc student working on the project received the second price of the National Scientific Student Conference. Csaba Domokos obtained his PhD degree and his work has been awarded the Attila Kuba Prize of the Hungarian Association for Image Processing and Pattern Recognition. More details about our results can be found at: • http://www.inf.u-szeged.hu/ipcg/projects/AFFSHAPE.html • http://www.inf.u-szeged.hu/ipcg/projects/AffinePuzzle.html • http://www.inf.u-szeged.hu/ipcg/projects/diffeoshape.htm

Registration Combining Wide and Narrow Baseline Feature Tracking Techniques for Markerless AR Systems

Augmented reality (AR) is a field of computer research which deals with the combination of real world and computer generated data. Registration is one of the most difficult problems currently limiting the usability of AR systems. In this paper, we propose a novel natural feature tracking based registration method for AR applications. The proposed method has following advantages: (1) it is simple and efficient, as no man-made markers are needed for both indoor and outdoor AR applications; moreover, it can work with arbitrary geometric shapes including planar, near planar and non planar structures which really enhance the usability of AR systems. (2) Thanks to the reduced SIFT based augmented optical flow tracker, the virtual scene can still be augmented on the specified areas even under the circumstances of occlusion and large changes in viewpoint during the entire process. (3) It is easy to use, because the adaptive classification tree based matching strategy can give us fast and accurate initialization, even when the initial camera is different from the reference image to a large degree. Experimental evaluations validate the performance of the proposed method for online pose tracking and augmentation

Robust Estimation of Motion Parameters and Scene Geometry : Minimal Solvers and Convexification of Regularisers for Low-Rank Approximation

In the dawning age of autonomous driving, accurate and robust tracking of vehicles is a quintessential part. This is inextricably linked with the problem of Simultaneous Localisation and Mapping (SLAM), in which one tries to determine the position of a vehicle relative to its surroundings without prior knowledge of them. The more you know about the object you wish to track—through sensors or mechanical construction—the more likely you are to get good positioning estimates. In the first part of this thesis, we explore new ways of improving positioning for vehicles travelling on a planar surface. This is done in several different ways: first, we generalise the work done for monocular vision to include two cameras, we propose ways of speeding up the estimation time with polynomial solvers, and we develop an auto-calibration method to cope with radially distorted images, without enforcing pre-calibration procedures.We continue to investigate the case of constrained motion—this time using auxiliary data from inertial measurement units (IMUs) to improve positioning of unmanned aerial vehicles (UAVs). The proposed methods improve the state-of-the-art for partially calibrated cases (with unknown focal length) for indoor navigation. Furthermore, we propose the first-ever real-time compatible minimal solver for simultaneous estimation of radial distortion profile, focal length, and motion parameters while utilising the IMU data.In the third and final part of this thesis, we develop a bilinear framework for low-rank regularisation, with global optimality guarantees under certain conditions. We also show equivalence between the linear and the bilinear framework, in the sense that the objectives are equal. This enables users of alternating direction method of multipliers (ADMM)—or other subgradient or splitting methods—to transition to the new framework, while being able to enjoy the benefits of second order methods. Furthermore, we propose a novel regulariser fusing two popular methods. This way we are able to combine the best of two worlds by encouraging bias reduction while enforcing low-rank solutions

Stereo vision without the scene-smoothness assumption: the homography-based approach.

by Andrew L. Arengo.Thesis (M.Phil.)--Chinese University of Hong Kong, 1998.Includes bibliographical references (leaves 65-66).Abstract also in Chinese.Acknowledgments --- p.iiList Of Figures --- p.vAbstract --- p.viiChapter 1 --- Introduction --- p.1Chapter 1.1 --- Motivation and Objective --- p.2Chapter 1.2 --- Approach of This Thesis and Contributions --- p.3Chapter 1.3 --- Organization of This Thesis --- p.4Chapter 2 --- Previous Work --- p.6Chapter 2.1 --- Using Grouped Features --- p.6Chapter 2.2 --- Applying Additional Heuristics --- p.7Chapter 2.3 --- Homography and Related Works --- p.9Chapter 3 --- Theory and Problem Formulation --- p.10Chapter 3.1 --- Overview of the Problems --- p.10Chapter 3.1.1 --- Preprocessing --- p.10Chapter 3.1.2 --- Establishing Correspondences --- p.11Chapter 3.1.3 --- Recovering 3D Depth --- p.14Chapter 3.2 --- Solving the Correspondence Problem --- p.15Chapter 3.2.1 --- Epipolar Constraint --- p.15Chapter 3.2.2 --- Surface-Continuity and Feature-Ordering Heuristics --- p.16Chapter 3.2.3 --- Using the Concept of Homography --- p.18Chapter 3.3 --- Concept of Homography --- p.20Chapter 3.3.1 --- Barycentric Coordinate System --- p.20Chapter 3.3.2 --- Image to Image Mapping of the Same Plane --- p.22Chapter 3.4 --- Problem Formulation --- p.23Chapter 3.4.1 --- Preliminaries --- p.23Chapter 3.4.2 --- Case of Single Planar Surface --- p.24Chapter 3.4.3 --- Case of Multiple Planar Surfaces --- p.28Chapter 3.5 --- Subspace Clustering --- p.28Chapter 3.6 --- Overview of the Approach --- p.30Chapter 4 --- Experimental Results --- p.33Chapter 4.1 --- Synthetic Images --- p.33Chapter 4.2 --- Aerial Images --- p.36Chapter 4.2.1 --- T-shape building --- p.38Chapter 4.2.2 --- Rectangular Building --- p.39Chapter 4.2.3 --- 3-layers Building --- p.40Chapter 4.2.4 --- Pentagon --- p.44Chapter 4.3 --- Indoor Scenes --- p.52Chapter 4.3.1 --- Stereo Motion Pair --- p.53Chapter 4.3.2 --- Hallway Scene --- p.56Chapter 5 --- Summary and Conclusions --- p.6

Geometry-Aware Network for Non-Rigid Shape Prediction from a Single View

We propose a method for predicting the 3D shape of a deformable surface from a single view. By contrast with previous approaches, we do not need a pre-registered template of the surface, and our method is robust to the lack of texture and partial occlusions. At the core of our approach is a {\it geometry-aware} deep architecture that tackles the problem as usually done in analytic solutions: first perform 2D detection of the mesh and then estimate a 3D shape that is geometrically consistent with the image. We train this architecture in an end-to-end manner using a large dataset of synthetic renderings of shapes under different levels of deformation, material properties, textures and lighting conditions. We evaluate our approach on a test split of this dataset and available real benchmarks, consistently improving state-of-the-art solutions with a significantly lower computational time.Comment: Accepted at CVPR 201

Augmented reality for non-rigid surfaces

Augmented Reality (AR) is the process of integrating virtual elements in reality, often by mixing computer graphics into a live video stream of a real scene. It requires registration of the target object with respect to the cameras. To this end, some approaches rely on dedicated hardware, such as magnetic trackers or infra-red cameras, but they are too expensive and cumbersome to reach a large public. Others are based on specifically designed markers which usually look like bar-codes. However, they alter the look of objects to be augmented, thereby hindering their use in application for which visual design matters. Recent advances in Computer Vision have made it possible to track and detect objects by relying on natural features. However, no such method is commonly used in the AR community, because the maturity of available packages is not sufficient yet. As far as deformable surfaces are concerned, the choice is even more limited, mainly because initialization is so difficult. Our main contribution is therefore a new AR framework that can properly augment deforming surfaces in real-time. Its target platform is a standard PC and a single webcam. It does not require any complex calibration procedure, making it perfectly suitable for novice end-users. To satisfy to the most demanding application designers, our framework does not require any scene engineering, renders virtual objects illuminated by real light, and let real elements occlude virtual ones. To meet this challenge, we developed several innovative techniques. Our approach to real-time registration of a deforming surface is based on wide-baseline feature matching. However, traditional outlier elimination techniques such as RANSAC are unable to handle the non-rigid surface's large number of degrees of freedom. We therefore proposed a new robust estimation scheme that allows both 2–D and 3–D non-rigid surface registration. Another issue of critical importance in AR to achieve realism is illumination handling, for which existing techniques often require setup procedures or devices such as reflective spheres. By contrast, our framework includes methods to estimate illumination for rendering purposes without sacrificing ease of use. Finally, several existing approaches to handling occlusions in AR rely on multiple cameras or can only deal with occluding objects modeled beforehand. Our requires only one camera and models occluding objects at runtime. We incorporated these components in a consistent and flexible framework. We used it to augment many different objects such as a deforming T-shirt or a sheet of paper, under challenging conditions, in real-time, and with correct handling of illumination and occlusions. We also used our non-rigid surface registration technique to measure the shape of deformed sails. We validated the ease of deployment of our framework by distributing a software package and letting an artist use it to create two AR applications

Super-Resolution Overlay in Multi-Projector Displays

A technique, associated system and computer executable program code, for projecting a superimposed image onto a target display surface under observation of one or more cameras. A projective relationship between each projector being used and the target display surface is determined using a suitable calibration technique. A component image for each projector is then estimated using the information from the calibration, and represented in the frequency domain. Each component image is estimated by: Using the projective relationship, determine a set of sub-sampled, regionally shifted images, represented in the frequency domain; each component image is then composed of a respective set of the sub-sampled, regionally shifted images. In an optimization step, the difference between a sum of the component images and a frequency domain representation of a target image is minimized to produce a second, or subsequent, component image for each projector

Content based image pose manipulation

This thesis proposes the application of space-frequency transformations to the domain of pose estimation in images. This idea is explored using the Wavelet Transform with illustrative applications in pose estimation for face images, and images of planar scenes. The approach is based on examining the spatial frequency components in an image, to allow the inherent scene symmetry balance to be recovered. For face images with restricted pose variation (looking left or right), an algorithm is proposed to maximise this symmetry in order to transform the image into a fronto-parallel pose. This scheme is further employed to identify the optimal frontal facial pose from a video sequence to automate facial capture processes. These features are an important pre-requisite in facial recognition and expression classification systems. The under lying principles of this spatial-frequency approach are examined with respect to images with planar scenes. Using the Continuous Wavelet Transform, full perspective planar transformations are estimated within a featureless framework. Restoring central symmetry to the wavelet transformed images in an iterative optimisation scheme removes this perspective pose. This advances upon existing spatial approaches that require segmentation and feature matching, and frequency only techniques that are limited to affine transformation recovery. To evaluate the proposed techniques, the pose of a database of subjects portraying varying yaw orientations is estimated and the accuracy is measured against the captured ground truth information. Additionally, full perspective homographies for synthesised and imaged textured planes are estimated. Experimental results are presented for both situations that compare favourably with existing techniques in the literature