HoughNet: neural network architecture for vanishing points detection

In this paper we introduce a novel neural network architecture based on Fast Hough Transform layer. The layer of this type allows our neural network to accumulate features from linear areas across the entire image instead of local areas. We demonstrate its potential by solving the problem of vanishing points detection in the images of documents. Such problem occurs when dealing with camera shots of the documents in uncontrolled conditions. In this case, the document image can suffer several specific distortions including projective transform. To train our model, we use MIDV-500 dataset and provide testing results. The strong generalization ability of the suggested method is proven with its applying to a completely different ICDAR 2011 dewarping contest. In previously published papers considering these dataset authors measured the quality of vanishing point detection by counting correctly recognized words with open OCR engine Tesseract. To compare with them, we reproduce this experiment and show that our method outperforms the state-of-the-art result.Comment: 6 pages, 6 figures, 2 tables, 28 references, conferenc

Generic Document Image Dewarping by Probabilistic Discretization of Vanishing Points

International audienceDocument images dewarping is still a challenge especially when documents are captured with one camera in an uncontrolled environment. In this paper we propose a generic approach based on vanishing points (VP) to reconstruct the 3D shape of document pages. Unlike previous methods we do not need to segment the text included in the documents. Therefore, our approach is less sensitive to pre-processing and segmentation errors. The computation of the VPs is robust and relies on the a-contrario framework, which has only one parameter whose setting is based on probabilistic reasoning instead of experimental tuning. Thus, our method can be applied to any kind of document including text and non-text blocks and extended to other kind of images. Experimental results show that the proposed method is robust to a variety of distortions

정렬 특성들 기반의 문서 및 장면 텍스트 영상 평활화 기법

학위논문 (박사)-- 서울대학교 대학원 공과대학 전기·컴퓨터공학부, 2017. 8. 조남익.카메라로 촬영한 텍스트 영상에 대해서, 광학 문자 인식(OCR)은 촬영된 장면을 분석하는데 있어서 매우 중요하다. 하지만 올바른 텍스트 영역 검출 후에도, 촬영한 영상에 대한 문자 인식은 여전히 어려운 문제로 여겨진다. 이는 종이의 구부러짐과 카메라 시점에 의한 기하학적인 왜곡 때문이고, 따라서 이러한 텍스트 영상에 대한 평활화는 문자 인식에 있어서 필수적인 전처리 과정으로 여겨진다. 이를 위한 왜곡된 촬영 영상을 정면 시점으로 복원하는 텍스트 영상 평활화 방법들은 활발히 연구되어지고 있다. 최근에는, 평활화가 잘 된 텍스트의 성질에 초점을 맞춘 연구들이 주로 진행되고 있다. 이러한 관점에서, 본 학위 논문은 텍스트 영상 평활화를 위하여 새로운 정렬 특성들을 다룬다. 이러한 정렬 특성들은 비용 함수로 설계되어지고, 비용 함수를 최소화하는 방법을 통해서 평활화에 사용되어지는 평활화 변수들이 구해진다. 본 학위 논문은 문서 영상 평활화, 장면 텍스트 평활화, 일반 배경 속의 휘어진 표면 평활화와 같이 3가지 세부 주제로 나눠진다. 첫 번째로, 본 학위 논문은 텍스트 라인들과 선분들의 정렬 특성에 기반의 문서 영상 평활화 방법을 제안한다. 기존의 텍스트 라인 기반의 문서 영상 평활화 방법들의 경우, 문서가 복잡한 레이아웃 형태이거나 적은 수의 텍스트 라인을 포함하고 있을 때 문제가 발생한다. 이는 문서에 텍스트 대신 그림, 그래프 혹은 표와 같은 영역이 많은 경우이다. 따라서 레이아웃에 강인한 문서 영상 평활화를 위하여 제안하는 방법은 정렬된 텍스트 라인뿐만 아니라 선분들도 이용한다. 올바르게 평활화 된 선분들은 여전히 일직선의 형태이고, 대부분 가로 혹은 세로 방향으로 정렬되어 있다는 가정 및 관측에 근거하여, 제안하는 방법은 이러한 성질들을 수식화하고 이를 텍스트 라인 기반의 비용 함수와 결합한다. 그리고 비용 함수를 최소화 하는 방법을 통해, 제안하는 방법은 종이의 구부러짐, 카메라 시점, 초점 거리와 같은 평활화 변수들을 추정한다. 또한, 오검출된 텍스트 라인들과 임의의 방향을 가지는 선분들과 같은 이상점(outlier)을 고려하여, 제안하는 방법은 반복적인 단계로 설계된다. 각 단계에서, 정렬 특성을 만족하지 않는 이상점들은 제거되고, 제거되지 않은 텍스트 라인 및 선분들만이 비용함수 최적화에 이용된다. 수행한 실험 결과들은 제안하는 방법이 다양한 레이아웃에 대하여 강인함을 보여준다. 두 번째로는, 본 논문은 장면 텍스트 평활화 방법을 제안한다. 기존 장면 텍스트 평활화 방법들의 경우, 가로/세로 방향의 획, 대칭 형태와 같은 문자가 가지는 고유의 생김새에 관련된 특성을 이용한다. 하지만, 이러한 방법들은 문자들의 정렬 형태는 고려하지 않고, 각각 개별 문자에 대한 특성들만을 이용하기 때문에 여러 문자들로 구성된 텍스트에 대해서 잘 정렬되지 않은 결과를 출력한다. 이러한 문제점을 해결하기 위하여, 제안하는 방법은 문자들의 정렬 정보를 이용한다. 정확하게는, 문자 고유의 모양뿐만 아니라 정렬 특성들도 함께 비용함수로 수식화되고, 비용함수를 최소화하는 방법을 통해서 평활화가 진행된다. 또한, 문자들의 정렬 특성을 수식화하기 위하여, 제안하는 방법은 텍스트를 각각 개별 문자들로 분리하는 문자 분리 또한 수행한다. 그 뒤, 텍스트의 위, 아래 선들을 RANSAC 알고리즘을 이용한 최소 제곱법을 통해 추정한다. 즉, 전체 알고리즘은 문자 분리와 선 추정, 평활화가 반복적으로 수행된다. 제안하는 비용함수는 볼록(convex)형태가 아니고 또한 많은 변수들을 포함하고 있기 때문에, 이를 최적화하기 위하여 Augmented Lagrange Multiplier 방법을 이용한다. 제안하는 방법은 일반 촬영 영상과 합성된 텍스트 영상을 통해 실험이 진행되었고, 실험 결과들은 제안하는 방법이 기존 방법들에 비하여 높은 인식 성능을 보이면서 동시에 시각적으로도 좋은 결과를 보임을 보여준다. 마지막으로, 제안하는 방법은 일반 배경 속의 휘어진 표면 평활화 방법으로도 확장된다. 일반 배경에 대해서, 약병이나 음료수 캔과 같이 원통 형태의 물체는 많이 존재한다. 그들의 표면은 일반 원통 표면(GCS)으로 모델링이 가능하다. 이러한 휘어진 표면들은 많은 문자와 그림들을 포함하고 있지만, 포함된 문자는 문서에 비해서 매우 불규칙적인 구조를 가지고 있다. 따라서 기존의 문서 영상 평활화 방법들로는 일반 배경 속 휘어진 표면 영상을 평활화하기 힘들다. 많은 휘어진 표면은 잘 정렬된 선분들 (테두리 선 혹은 바코드)을 포함하고 있다는 관측에 근거하여, 제안하는 방법은 앞서 제안한 선분들에 대한 함수를 이용하여 휘어진 표면을 평활화한다. 다양한 둥근 물체의 휘어진 표면 영상들에 대한 실험 결과들은 제안하는 방법이 평활화를 정확하게 수행함을 보여준다.The optical character recognition (OCR) of text images captured by cameras plays an important role for scene understanding. However, the OCR of camera-captured image is still considered a challenging problem, even after the text detection (localization). It is mainly due to the geometric distortions caused by page curve and perspective view, therefore their rectification has been an essential pre-processing step for their recognition. Thus, there have been many text image rectification methods which recover the fronto-parallel view image from a single distorted image. Recently, many researchers have focused on the properties of the well-rectified text. In this respect, this dissertation presents novel alignment properties for text image rectification, which are encoded into the proposed cost functions. By minimizing the cost functions, the transformation parameters for rectification are obtained. In detail, they are applied to three topics: document image dewarping, scene text rectification, and curved surface dewarping in real scene. First, a document image dewarping method is proposed based on the alignments of text-lines and line segments. Conventional text-line based document dewarping methods have problems when handling complex layout and/or very few text-lines. When there are few aligned text-lines in the image, this usually means that photos, graphics and/or tables take large portion of the input instead. Hence, for the robust document dewarping, the proposed method uses line segments in the image in addition to the aligned text-lines. Based on the assumption and observation that all the transformed line segments are still straight (line to line mapping), and many of them are horizontally or vertically aligned in the well-rectified images, the proposed method encodes this properties into the cost function in addition to the text-line based cost. By minimizing the function, the proposed method can obtain transformation parameters for page curve, camera pose, and focal length, which are used for document image rectification. Considering that there are many outliers in line segment directions and miss-detected text-lines in some cases, the overall algorithm is designed in an iterative manner. At each step, the proposed method removes the text-lines and line segments that are not well aligned, and then minimizes the cost function with the updated information. Experimental results show that the proposed method is robust to the variety of page layouts. This dissertation also presents a method for scene text rectification. Conventional methods for scene text rectification mainly exploited the glyph property, which means that the characters in many language have horizontal/vertical strokes and also some symmetric shapes. However, since they consider the only shape properties of individual character, without considering the alignments of characters, they work well for only images with a single character, and still yield mis-aligned results for images with multiple characters. In order to alleviate this problem, the proposed method explicitly imposes alignment constraints on rectified results. To be precise, character alignments as well as glyph properties are encoded in the proposed cost function, and the transformation parameters are obtained by minimizing the function. Also, in order to encode the alignments of characters into the cost function, the proposed method separates the text into individual characters using a projection profile method before optimizing the cost function. Then, top and bottom lines are estimated using a least squares line fitting with RANSAC. Overall algorithm is designed to perform character segmentation, line fitting, and rectification iteratively. Since the cost function is non-convex and many variables are involved in the function, the proposed method also develops an optimization method using Augmented Lagrange Multiplier method. This dissertation evaluates the proposed method on real and synthetic text images and experimental results show that the proposed method achieves higher OCR accuracy than the conventional approach and also yields visually pleasing results. Finally, the proposed method can be extended to the curved surface dewarping in real scene. In real scene, there are many circular objects such as medicine bottles or cans of drinking water, and their curved surfaces can be modeled as Generalized Cylindrical Surfaces (GCS). These curved surfaces include many significant text and figures, however their text has irregular structure compared to documents. Therefore, the conventional dewarping methods based on the properties of well-rectified text have problems in their rectification. Based on the observation that many curved surfaces include well-aligned line segments (boundary lines of objects or barcode), the proposed method rectifies the curved surfaces by exploiting the proposed line segment terms. Experimental results on a range of images with curved surfaces of circular objects show that the proposed method performs rectification robustly.1 Introduction 1 1.1 Document image dewarping 3 1.2 Scene text rectification 5 1.3 Curved surface dewarping in real scene 7 1.4 Contents 8 2 Related work 9 2.1 Document image dewarping 9 2.1.1 Dewarping methods using additional information 9 2.1.2 Text-line based dewarping methods 10 2.2 Scene text rectification 11 2.3 Curved surface dewarping in real scene 12 3 Document image dewarping 15 3.1 Proposed cost function 15 3.1.1 Parametric model of dewarping process 15 3.1.2 Cost function design 18 3.1.3 Line segment properties and cost function 19 3.2 Outlier removal and optimization 26 3.2.1 Jacobian matrix of the proposed cost function 27 3.3 Document region detection and dewarping 31 3.4 Experimental results 32 3.4.1 Experimental results on text-abundant document images 33 3.4.2 Experimental results on non conventional document images 34 3.5 Summary 47 4 Scene text rectification 49 4.1 Proposed cost function for rectification 49 4.1.1 Cost function design 49 4.1.2 Character alignment properties and alignment terms 51 4.2 Overall algorithm 54 4.2.1 Initialization 55 4.2.2 Character segmentation 56 4.2.3 Estimation of the alignment parameters 57 4.2.4 Cost function optimization for rectification 58 4.3 Experimental results 63 4.4 Summary 66 5 Curved surface dewarping in real scene 73 5.1 Proposed curved surface dewarping method 73 5.1.1 Pre-processing 73 5.1 Experimental results 74 5.2 Summary 76 6 Conclusions 83 Bibliography 85 Abstract (Korean) 93Docto

An investigation into common challenges of 3D scene understanding in visual surveillance

Nowadays, video surveillance systems are ubiquitous. Most installations simply consist of CCTV cameras connected to a central control room and rely on human operators to interpret what they see on the screen in order to, for example, detect a crime (either during or after an event). Some modern computer vision systems aim to automate the process, at least to some degree, and various algorithms have been somewhat successful in certain limited areas. However, such systems remain inefficient in general circumstances and present real challenges yet to be solved. These challenges include the ability to recognise and ultimately predict and prevent abnormal behaviour or even reliably recognise objects, for example in order to detect left luggage or suspicious objects. This thesis first aims to study the state-of-the-art and identify the major challenges and possible requirements of future automated and semi-automated CCTV technology in the field. This thesis presents the application of a suite of 2D and highly novel 3D methodologies that go some way to overcome current limitations.The methods presented here are based on the analysis of object features directly extracted from the geometry of the scene and start with a consideration of mainly existing techniques, such as the use of lines, vanishing points (VPs) and planes, applied to real scenes. Then, an investigation is presented into the use of richer 2.5D/3D surface normal data. In all cases the aim is to combine both 2D and 3D data to obtain a better understanding of the scene, aimed ultimately at capturing what is happening within the scene in order to be able to move towards automated scene analysis. Although this thesis focuses on the widespread application of video surveillance, an example case of the railway station environment is used to represent typical real-world challenges, where the principles can be readily extended elsewhere, such as to airports, motorways, the households, shopping malls etc. The context of this research work, together with an overall presentation of existing methods used in video surveillance and their challenges are described in chapter 1.Common computer vision techniques such as VP detection, camera calibration, 3D reconstruction, segmentation etc., can be applied in an effort to extract meaning to video surveillance applications. According to the literature, these methods have been well researched and their use will be assessed in the context of current surveillance requirements in chapter 2. While existing techniques can perform well in some contexts, such as an architectural environment composed of simple geometrical elements, their robustness and performance in feature extraction and object recognition tasks is not sufficient to solve the key challenges encountered in general video surveillance context. This is largely due to issues such as variable lighting, weather conditions, and shadows and in general complexity of the real-world environment. Chapter 3 presents the research and contribution on those topics – methods to extract optimal features for a specific CCTV application – as well as their strengths and weaknesses to highlight that the proposed algorithm obtains better results than most due to its specific design.The comparison of current surveillance systems and methods from the literature has shown that 2D data are however almost constantly used for many applications. Indeed, industrial systems as well as the research community have been improving intensively 2D feature extraction methods since image analysis and Scene understanding has been of interest. The constant progress on 2D feature extraction methods throughout the years makes it almost effortless nowadays due to a large variety of techniques. Moreover, even if 2D data do not allow solving all challenges in video surveillance or other applications, they are still used as starting stages towards scene understanding and image analysis. Chapter 4 will then explore 2D feature extraction via vanishing point detection and segmentation methods. A combination of most common techniques and a novel approach will be then proposed to extract vanishing points from video surveillance environments. Moreover, segmentation techniques will be explored in the aim to determine how they can be used to complement vanishing point detection and lead towards 3D data extraction and analysis. In spite of the contribution above, 2D data is insufficient for all but the simplest applications aimed at obtaining an understanding of a scene, where the aim is for a robust detection of, say, left luggage or abnormal behaviour; without significant a priori information about the scene geometry. Therefore, more information is required in order to be able to design a more automated and intelligent algorithm to obtain richer information from the scene geometry and so a better understanding of what is happening within. This can be overcome by the use of 3D data (in addition to 2D data) allowing opportunity for object “classification” and from this to infer a map of functionality, describing feasible and unfeasible object functionality in a given environment. Chapter 5 presents how 3D data can be beneficial for this task and the various solutions investigated to recover 3D data, as well as some preliminary work towards plane extraction.It is apparent that VPs and planes give useful information about a scene’s perspective and can assist in 3D data recovery within a scene. However, neither VPs nor plane detection techniques alone allow the recovery of more complex generic object shapes - for example composed of spheres, cylinders etc - and any simple model will suffer in the presence of non-Manhattan features, e.g. introduced by the presence of an escalator. For this reason, a novel photometric stereo-based surface normal retrieval methodology is introduced to capture the 3D geometry of the whole scene or part of it. Chapter 6 describes how photometric stereo allows recovery of 3D information in order to obtain a better understanding of a scene, as well as also partially overcoming some current surveillance challenges, such as difficulty in resolving fine detail, particularly at large standoff distances, and in isolating and recognising more complex objects in real scenes. Here items of interest may be obscured by complex environmental factors that are subject to rapid change, making, for example, the detection of suspicious objects and behaviour highly problematic. Here innovative use is made of an untapped latent capability offered within modern surveillance environments to introduce a form of environmental structuring to good advantage in order to achieve a richer form of data acquisition. This chapter also goes on to explore the novel application of photometric stereo in such diverse applications, how our algorithm can be incorporated into an existing surveillance system and considers a typical real commercial application.One of the most important aspects of this research work is its application. Indeed, while most of the research literature has been based on relatively simple structured environments, the approach here has been designed to be applied to real surveillance environments, such as railway stations, airports, waiting rooms, etc, and where surveillance cameras may be fixed or in the future form part of a mobile robotic free roaming surveillance device, that must continually reinterpret its changing environment. So, as mentioned previously, while the main focus has been to apply this algorithm to railway station environments, the work has been approached in a way that allows adaptation to many other applications, such as autonomous robotics, and in motorway, shopping centre, street and home environments. All of these applications require a better understanding of the scene for security or safety purposes. Finally, chapter 7 presents a global conclusion and what will be achieved in the future

Ajuste de Perspectiva Automático Aplicado em Imagens de Gôndolas de Supermercado

TCC(graduação) - Universidade Federal de Santa Catarina. Campus Araranguá. Engenharia da Computação.Grande parte dos produtos presentes nas gôndolas de estabelecimentos comerciais não são notadas pelos consumidores. Por esse motivo, a indústria necessita criar e monitorar as estratégias utilizadas nos pontos de venda para aumentar o alcance das ações de marketing. Isso pode ser planejado utilizando fotos da gôndola dos locais, obtidas por promotores de vendas, e processadas a posteriori por métodos de processamento digital de imagens e reconhecimento de padrões. Entretanto, caso não for possível capturar uma imagem perpendicular ao plano frontal, pode haver perda de informações dos produtos apresentados na mesma. Nesse contexto, o trabalho em questão apresenta a modelagem de uma solução computacional totalmente automática cujo objetivo é realizar a correção da perspectiva presente em imagens de gôndolas de supermercados. Utilizando métodos computacionais já consolidados na literatura, baseados em pontos de fuga, foi possível obter um alinhamento adequado em 94% das imagens ajustadas e, em um dataset de validação, comprovou-se que as fotos corrigidas ficaram mais semelhantes às suas fotos frontais em todos os casos testados.Most of the products present on the shelves of commercial establishments are not noticed by consumers. For this reason, the industry needs to create and monitor the strategies used at points of sale to increase the reach of marketing actions. This can be planned using photos of the shelf, obtained by sales promoters and processed a posteriori by digital image processing and pattern recognition methods. However, if it is not possible to capture an image perpendicular to the frontal plane, there may be loss of information about the products presented on the shelf. In this context, the current study presents the modeling of a fully automatic computational solution whose goal is to perform the correction of the perspective inserted in supermarket shelf images. Using computational methods already consolidated in the literature, based on vanishing points, it was possible to obtain a proper alignment in 94% of the adjusted images and, in a validation dataset, it was verified that the corrected photos were more similar to their frontal photos in all cases tested

Reversing ShopView analysis for planogram creation

Com o aumento da preocupação dos retalhistas na melhora dos resultados das vendas e da experiência dos consumidores, existe uma necessidade de desenvolver tecnologia que auxilie a otimização desses objetivos. Está provado que uma correta colocação dos produtos nas prateleiras pode aumentar significativamente as vendas e a experiência dos consumidores [1]. Com isto em mente, a Fraunhofer Portugal desenvolveu o ShopView [2], uma solução que tem como objetivo ajudar os retalhistas a extrair, validar e manipular os planogramas a partir de imagens de alta resolução das lojas. Desta forma, esta tese tem como foco a criação de um algoritmo, com auxilio a algoritmos de visão computacional, de forma a extrair informação de images de alta resolução de gondolas de supermercados obtidas pelo ShopView. Particularmente, foram implementados passos de pré-processamento de forma a melhorara a eficiência e precisão de um motor de OCR (Optical Character Recognition) no reconhecimento de texto nos produtos da gondola. Estes algoritmos de pré-processamento são compostos por técnicas de segmentação e remoção de ruido. O uso deste motor de OCR permite obter informações adicionais sobre os produtos e separações das prateleiras, esta informação é posteriormente usada em algoritmos de agrupamento para extrair automáticamente um planograma preciso a partir das imagens. O algoritmo apresentado é capaz de extrair informação relevante das imagens das gondolas, de forma a identificar os produtos existentes e criar metadados válidos sobre esses produtos e a sua localização. Com estes metadados é possível criar, validar e modificar o planograma no ShopView. O uso de OCR neste algoritmo oferece vantagens sobre as outras abordagens disponíveis devido à capacidade de diferenciar produtos com diferenças mínimas entre si, e a sua imunidade a alterações no aspeto das embalagens dos produtos. Além disso, a metodologia proposta não requer nenhuma interação prévia do utilizador para funcionar corretamente.[1] Chanjin Chung, Todd M. Schmit, Diansheng Dong, and Harry M. Kaiser. Economic evaluation of shelf-space management in grocery stores. Agribusiness, 23(4):583-597, September 2007.[2] L. Rosado, J. Gonçalves, J. Costa, D. Ribeiro, and F. Soares. Supervised learning for out-of-stock detection in panoramas of retail shelves. In 2016 IEEE International Conference on Imaging Systems and Techniques (IST), pages 406-411.With the increasing care of retail shop owners in improving sales and costumer experience, there is a need to develop technology in order to optimize their goals. It's proven that a planned product placement can boost sales and improve costumer experience [1]. With this in mind, Fraunhofer Portugal came up with ShopView [2] solution to help retail shops extract, validate and manipulate planograms from high-resolution images of the real shelves in the store.In this sense, this thesis focused on the creation of an algorithm using computer vision algorithms to extract information from high resolution images of retail shelves taken with ShopView solution. Particularly, it was implemented pre-processing steps to improve the efficiency and accuracy of an OCR (Optical Character Recognition) engine in recognizing the text in the shelves products. These pre-processing algorithms comprise of denoising and segmentation techniques. The use of this OCR engine brings additional information about products and shelves separation, this information is later used in clustering algorithms to automatically extract an accurate planogram from shelves photos. The presented algorithm is capable of extracting relevant information from the shelves images, to identify the existing products and create a valid metadata about them and their location. With this metadata, it is possible to create, validate and modify the planogram in ShopView. The use of OCR on this algorithm has advantages over other available approaches due to its capability to differentiate products with minimal visual differences and its immunity to appearance changes on the products packaging. Moreover, the methodology proposed does not require any previous user interaction to work properly.[1] Chanjin Chung, Todd M. Schmit, Diansheng Dong, and Harry M. Kaiser. Economic evaluation of shelf-space management in grocery stores. Agribusiness, 23(4):583-597, September 2007.[2] L. Rosado, J. Gonçalves, J. Costa, D. Ribeiro, and F. Soares. Supervised learning for out-of-stock detection in panoramas of retail shelves. In 2016 IEEE International Conference on Imaging Systems and Techniques (IST), pages 406-411

Airborne vision-based attitude estimation and localisation

Vision plays an integral part in a pilot's ability to navigate and control an aircraft. Therefore Visual Flight Rules have been developed around the pilot's ability to see the environment outside of the cockpit in order to control the attitude of the aircraft, to navigate and to avoid obstacles. The automation of these processes using a vision system could greatly increase the reliability and autonomy of unmanned aircraft and flight automation systems. This thesis investigates the development and implementation of a robust vision system which fuses inertial information with visual information in a probabilistic framework with the aim of aircraft navigation. The horizon appearance is a strong visual indicator of the attitude of the aircraft. This leads to the first research area of this thesis, visual horizon attitude determination. An image processing method was developed to provide high performance horizon detection and extraction from camera imagery. A number of horizon models were developed to link the detected horizon to the attitude of the aircraft with varying degrees of accuracy. The second area investigated in this thesis was visual localisation of the aircraft. A terrain-aided horizon model was developed to estimate the position, altitude as well as attitude of the aircraft. This gives rough positions estimates with highly accurate attitude information. The visual localisation accuracy was improved by incorporating ground feature-based map-aided navigation. Road intersections were detected using a developed image processing algorithm and then they were matched to a database to provide positional information. The developed vision system show comparable performance to other non-vision-based systems while removing the dependence on external systems for navigation. The vision system and techniques developed in this thesis helps to increase the autonomy of unmanned aircraft and flight automation systems for manned flight

Multi-scale data fusion for surface metrology

The major trends in manufacturing are miniaturization, convergence of the traditional research fields and creation of interdisciplinary research areas. These trends have resulted in the development of multi-scale models and multi-scale surfaces to optimize the performance. Multi-scale surfaces that exhibit specific properties at different scales for a specific purpose require multi-scale measurement and characterization. Researchers and instrument developers have developed instruments that are able to perform measurements at multiple scales but lack the much required multi- scale characterization capability. The primary focus of this research was to explore possible multi-scale data fusion strategies and options for surface metrology domain and to develop enabling software tools in order to obtain effective multi-scale surface characterization, maximizing fidelity while minimizing measurement cost and time. This research effort explored the fusion strategies for surface metrology domain and narrowed the focus on Discrete Wavelet Frame (DWF) based multi-scale decomposition. An optimized multi-scale data fusion strategy ‘FWR method’ was developed and was successfully demonstrated on both high aspect ratio surfaces and non-planar surfaces. It was demonstrated that the datum features can be effectively characterized at a lower resolution using one system (Vision CMM) and the actual features of interest could be characterized at a higher resolution using another system (Coherence Scanning Interferometer) with higher capability while minimizing the measurement time

Geometric Inference with Microlens Arrays

This dissertation explores an alternative to traditional fiducial markers where geometric information is inferred from the observed position of 3D points seen in an image. We offer an alternative approach which enables geometric inference based on the relative orientation of markers in an image. We present markers fabricated from microlenses whose appearance changes depending on the marker\u27s orientation relative to the camera. First, we show how to manufacture and calibrate chromo-coding lenticular arrays to create a known relationship between the observed hue and orientation of the array. Second, we use 2 small chromo-coding lenticular arrays to estimate the pose of an object. Third, we use 3 large chromo-coding lenticular arrays to calibrate a camera with a single image. Finally, we create another type of fiducial marker from lenslet arrays that encode orientation with discrete black and white appearances. Collectively, these approaches oer new opportunities for pose estimation and camera calibration that are relevant for robotics, virtual reality, and augmented reality