Depth recovery and parameter analysis using single-lens prism based stereovision system

Ph.DDOCTOR OF PHILOSOPH

Depth Recovery with Rectification using Single-Lens Prism based Stereovision System

Ph.DDOCTOR OF PHILOSOPH

Stereo Correspondence and Depth Recovery of Single-lens Bi-prism Based Stereovision System

Ph.DDOCTOR OF PHILOSOPH

Segment-based stereo matching algorithm with rectification for single-lens bi-prism stereovision system

Ph.DDOCTOR OF PHILOSOPH

Single-lens multi-ocular stereovision using prism

Ph.DDOCTOR OF PHILOSOPH

AN ALGORITHM FOR RECONSTRUCTING THREE-DIMENSIONAL IMAGES FROM OVERLAPPING TWO-DIMENSIONAL INTENSITY MEASUREMENTS WITH RELAXED CAMERA POSITIONING REQUIREMENTS, WITH APPLICATION TO ADDITIVE MANUFACTURING

Cameras are everywhere for security purposes and there are often many cameras installed close to each other to cover areas of interest, such as airport passenger terminals. These systems are often designed to have overlapping fields of view to provide different aspects of the scene to review when, for example, law enforcement issues arise. However, these cameras are rarely, if ever positioned in a way that would be conducive to conventional stereo image processing. To address this, issue an algorithm was developed to rectify images measured under such conditions, and then perform stereo image reconstruction. The initial experiments described here were set up using two scientific cameras to capture overlapping images in various cameras positons. The results showed that the algorithm was accurately reconstructing the three-dimensional (3-D) surface locations of the input objects. During the research an opportunity arose to further develop and test the algorithms for the problem of monitoring the fabrication process inside a 3-D printer. The geometry of 3-D printers prevents the location of cameras in the conventional stereo imaging geometry, making the algorithms described above seem like an attractive solution to this problem. The emphasis in 3-D printing on using extremely low cost components and open source software, and the need to develop the means of comparing observed progress in the fabrication process to a model of the device being fabricated posed additional development challenges. Inside the 3-D printer the algorithm was applied using two scientific cameras to detect the errors during the printing of the low-cost open-source RepRap style 3-D printer developed by the Michigan Tech’s Open Sustainability Technology Lab. An algorithm to detect errors in the shape of a device being fabricated using only one camera was also developed. The results show that a 3-D reconstruction algorithm can be used to accurately detect the 3-D printing errors. The initial development of the algorithm was in MATLAB. The cost of the MATLAB software might prevent it from being used by open-source communities. Thus, the algorithm was ported to Python and made open-source for everyone to use and customize. To reduce the cost, the commonly used and widely available inexpensive webcams were also used instead of the expensive scientific cameras. In order to detect errors around the printed part, six webcams were used, so there were 3 pairs of webcams and each pair were 120 degrees apart. The results indicated that the algorithms are precisely detect the 3-D printing errors around the printed part in shape and size aspects. With this low-cost and open-source approach, the algorithms are ready for wide range of use and applications

IMAGE DISTORTION CORRECTION FOR BIPRISM-BASED SINGLE-LENS STEREOVISION SYSTEM

Ph.DDOCTOR OF PHILOSOPH

Optimal calibration of a prism-based videoendoscopic system for precise 3D measurements

Modern videoendoscopes are capable of performing precise three-dimensional (3D) measurements of hard-to-reach elements. An attachable prism-based stereo adapter allows one to register images from two different viewpoints using a single sensor and apply stereoscopic methods. The key condition for achieving high measurement accuracy is the optimal choice of a mathematical model for calibration and 3D reconstruction procedures. In this paper, the conventional pinhole camera models with polynomial distortion approximation were analyzed and compared to the ray tracing model based on the vector form of Snell’s law. We, first, conducted a series of experiments using an industrial videoendoscope and utilized the criteria based on the measurement error of a segment length to evaluate the mathematical models considered. The experimental results confirmed a theoretical conclusion that the ray tracing model outperforms the pinhole models in a wide range of working distances. The results may be useful for the development of new stereoscopic measurement tools and algorithms for remote visual inspection in industrial and medical applications.The Russian Science Foundation (project #7-19-01355) financially supported the work. The authors thank A. Naumov, A. Shurygin and D. Khokhlov for continuous technical support

An investigation into common challenges of 3D scene understanding in visual surveillance

Nowadays, video surveillance systems are ubiquitous. Most installations simply consist of CCTV cameras connected to a central control room and rely on human operators to interpret what they see on the screen in order to, for example, detect a crime (either during or after an event). Some modern computer vision systems aim to automate the process, at least to some degree, and various algorithms have been somewhat successful in certain limited areas. However, such systems remain inefficient in general circumstances and present real challenges yet to be solved. These challenges include the ability to recognise and ultimately predict and prevent abnormal behaviour or even reliably recognise objects, for example in order to detect left luggage or suspicious objects. This thesis first aims to study the state-of-the-art and identify the major challenges and possible requirements of future automated and semi-automated CCTV technology in the field. This thesis presents the application of a suite of 2D and highly novel 3D methodologies that go some way to overcome current limitations.The methods presented here are based on the analysis of object features directly extracted from the geometry of the scene and start with a consideration of mainly existing techniques, such as the use of lines, vanishing points (VPs) and planes, applied to real scenes. Then, an investigation is presented into the use of richer 2.5D/3D surface normal data. In all cases the aim is to combine both 2D and 3D data to obtain a better understanding of the scene, aimed ultimately at capturing what is happening within the scene in order to be able to move towards automated scene analysis. Although this thesis focuses on the widespread application of video surveillance, an example case of the railway station environment is used to represent typical real-world challenges, where the principles can be readily extended elsewhere, such as to airports, motorways, the households, shopping malls etc. The context of this research work, together with an overall presentation of existing methods used in video surveillance and their challenges are described in chapter 1.Common computer vision techniques such as VP detection, camera calibration, 3D reconstruction, segmentation etc., can be applied in an effort to extract meaning to video surveillance applications. According to the literature, these methods have been well researched and their use will be assessed in the context of current surveillance requirements in chapter 2. While existing techniques can perform well in some contexts, such as an architectural environment composed of simple geometrical elements, their robustness and performance in feature extraction and object recognition tasks is not sufficient to solve the key challenges encountered in general video surveillance context. This is largely due to issues such as variable lighting, weather conditions, and shadows and in general complexity of the real-world environment. Chapter 3 presents the research and contribution on those topics – methods to extract optimal features for a specific CCTV application – as well as their strengths and weaknesses to highlight that the proposed algorithm obtains better results than most due to its specific design.The comparison of current surveillance systems and methods from the literature has shown that 2D data are however almost constantly used for many applications. Indeed, industrial systems as well as the research community have been improving intensively 2D feature extraction methods since image analysis and Scene understanding has been of interest. The constant progress on 2D feature extraction methods throughout the years makes it almost effortless nowadays due to a large variety of techniques. Moreover, even if 2D data do not allow solving all challenges in video surveillance or other applications, they are still used as starting stages towards scene understanding and image analysis. Chapter 4 will then explore 2D feature extraction via vanishing point detection and segmentation methods. A combination of most common techniques and a novel approach will be then proposed to extract vanishing points from video surveillance environments. Moreover, segmentation techniques will be explored in the aim to determine how they can be used to complement vanishing point detection and lead towards 3D data extraction and analysis. In spite of the contribution above, 2D data is insufficient for all but the simplest applications aimed at obtaining an understanding of a scene, where the aim is for a robust detection of, say, left luggage or abnormal behaviour; without significant a priori information about the scene geometry. Therefore, more information is required in order to be able to design a more automated and intelligent algorithm to obtain richer information from the scene geometry and so a better understanding of what is happening within. This can be overcome by the use of 3D data (in addition to 2D data) allowing opportunity for object “classification” and from this to infer a map of functionality, describing feasible and unfeasible object functionality in a given environment. Chapter 5 presents how 3D data can be beneficial for this task and the various solutions investigated to recover 3D data, as well as some preliminary work towards plane extraction.It is apparent that VPs and planes give useful information about a scene’s perspective and can assist in 3D data recovery within a scene. However, neither VPs nor plane detection techniques alone allow the recovery of more complex generic object shapes - for example composed of spheres, cylinders etc - and any simple model will suffer in the presence of non-Manhattan features, e.g. introduced by the presence of an escalator. For this reason, a novel photometric stereo-based surface normal retrieval methodology is introduced to capture the 3D geometry of the whole scene or part of it. Chapter 6 describes how photometric stereo allows recovery of 3D information in order to obtain a better understanding of a scene, as well as also partially overcoming some current surveillance challenges, such as difficulty in resolving fine detail, particularly at large standoff distances, and in isolating and recognising more complex objects in real scenes. Here items of interest may be obscured by complex environmental factors that are subject to rapid change, making, for example, the detection of suspicious objects and behaviour highly problematic. Here innovative use is made of an untapped latent capability offered within modern surveillance environments to introduce a form of environmental structuring to good advantage in order to achieve a richer form of data acquisition. This chapter also goes on to explore the novel application of photometric stereo in such diverse applications, how our algorithm can be incorporated into an existing surveillance system and considers a typical real commercial application.One of the most important aspects of this research work is its application. Indeed, while most of the research literature has been based on relatively simple structured environments, the approach here has been designed to be applied to real surveillance environments, such as railway stations, airports, waiting rooms, etc, and where surveillance cameras may be fixed or in the future form part of a mobile robotic free roaming surveillance device, that must continually reinterpret its changing environment. So, as mentioned previously, while the main focus has been to apply this algorithm to railway station environments, the work has been approached in a way that allows adaptation to many other applications, such as autonomous robotics, and in motorway, shopping centre, street and home environments. All of these applications require a better understanding of the scene for security or safety purposes. Finally, chapter 7 presents a global conclusion and what will be achieved in the future