Planar Prior Assisted PatchMatch Multi-View Stereo

The completeness of 3D models is still a challenging problem in multi-view stereo (MVS) due to the unreliable photometric consistency in low-textured areas. Since low-textured areas usually exhibit strong planarity, planar models are advantageous to the depth estimation of low-textured areas. On the other hand, PatchMatch multi-view stereo is very efficient for its sampling and propagation scheme. By taking advantage of planar models and PatchMatch multi-view stereo, we propose a planar prior assisted PatchMatch multi-view stereo framework in this paper. In detail, we utilize a probabilistic graphical model to embed planar models into PatchMatch multi-view stereo and contribute a novel multi-view aggregated matching cost. This novel cost takes both photometric consistency and planar compatibility into consideration, making it suited for the depth estimation of both non-planar and planar regions. Experimental results demonstrate that our method can efficiently recover the depth information of extremely low-textured areas, thus obtaining high complete 3D models and achieving state-of-the-art performance.Comment: Accepted by AAAI-202

Hopfield neural network for stereo matching

ln this paper, we present an algorithm designed for the stereovision matching problem and 3D identification. We use a simulated Hopfield neural network to solve the problem of matching a pair of stereoscopic images. This mode1 is helpful in optimization and it can be implemented on parallel machines easily.Nous nous intéressons au problème d'appariement de primitives entre deux images. Notre domaine d'application est la mise en correspondance d'un couple d'images stéréoscopiques ou l'identification des parties d'un modèle dans une image observée. Nous proposons dans ce papier une approche utilisant un modèle de réseau de neurones pour résoudre le problème. Nous avons choisi le modèle de Hopfield d'une part parce qu'il est souple et ouvert, d'autre part parce qu'il peut s'implanter aisément sur des calculateurs massivement parallèles

Stitching algorithms for biological specimen images

Abstract: In this paper, we address the problem of combining multiple overlapping image sections of biological specimens to obtain a single image containing the entire specimen. This is useful in the digitisation of a large number of biological specimens stored in museum collections and laboratories. In the case of many large specimens, it means that the specimen must be captured in overlapping sections instead of a single image. In this research, we have compared the performance of several known algorithms for this problem. In addition, we have developed several new algorithms based on matching the geometry (width, slope, and curvature) of the specimens at the boundaries. Finally, we compare the performance of a bagging approach that combines the results from multiple stitching algorithms. Our detailed evaluation shows that brightness-based and curvature-based approaches produce the best matches for the images in this domain

Multi-camera object segmentation in dynamically textured scenes using disparity contours

This thesis presents a stereo-based object segmentation system that combines the simplicity and efficiency of the background subtraction approach with the capacity of dealing with dynamic lighting and background texture and large textureless regions. The method proposed here does not rely on full stereo reconstruction or empirical parameter tuning, but employs disparity-based hypothesis verification to separate multiple objects at different depths.The proposed stereo-based segmentation system uses a pair of calibrated cameras with a small baseline and factors the segmentation problem into two stages: a well-understood offline stage and a novel online one. Based on the calibrated parameters, the offline stage models the 3D geometry of a background by constructing a complete disparity map. The online stage compares corresponding new frames synchronously captured by the two cameras according to the background disparity map in order to falsify the hypothesis that the scene contains only background. The resulting object boundary contours possess a number of useful features that can be exploited for object segmentation.Three different approaches to contour extraction and object segmentation were experimented with and their advantages and limitations analyzed. The system demonstrates its ability to extract multiple objects from a complex scene with near real-time performance. The algorithm also has the potential of providing precise object boundaries rather than just bounding boxes, and is extensible to perform 2D and 3D object tracking and online background update

Stereoskopische Korrespondenzbestimmung mit impliziter Detektion von Okklusionen

Der Einsatz binokularer Sehsysteme eröffnet sowohl in der Natur als auch in der Technik die Möglichkeit zum räumlichen Sehen.Das Grundprinzip bildet hierbei eine passive Triangulation, deren Ausgangspunkte die korrespondierenden Positionen darstellen, auf die ein Raumpunkt in die Stereobilder projiziert wird. Das zentrale Problem besteht bei dieser Technik darin, die korrespondierenden Bildpunkte eindeutig einander zuzuordnen. Dieses sogenannte Korrespondenzproblem ist einerseits aufgrund mehrerer ähnlicher Strukturen in der betrachteten Szene oft stark mehrdeutig und besitzt andererseits nicht immer eine Lösung, da Bereiche in der Szeneauftreten können, die nur aus einer der beiden Perspektiven zusehen sind. Weiterhin wird eine eindeutige Zuordnung korrespondierender Bildbereiche durch interokuläre Differenzen wie perspektivische Verzerrungen, Beleuchtungsunterschiede und Rauschprozesse zusätzlich erschwert. In der vorliegenden Arbeit werden die einzelnen Komponenten eines Gesamtsystems vorgestellt, die zur stereoskopischen Rekonstruktion der räumlichen Struktur einer Szene erforderlich sind. Den Schwerpunkt der Arbeit bildet ein Selbstorganisationsprozeß, der in Verbindung mit weiteren Verfahrensschritten eine eindeutige Zuordnung korrespondierender Bildpunkte erlaubt. Darüber hinaus werden hierbei einseitig sichtbare Bildbereiche, die eine wesentliche Fehlerursache in der Stereoskopie darstellen, detektiert und vom Zuordnungsprozeß ausgeschlossen.Stereo vision is a passive method used to recover the depth information of a scene, which is lost during the projection of a point in the 3D-scene onto the 2D image plane. In stereo vision, in which two or more views of a scene are used, the depth information can be reconstructed from the different positions in the images to which a physical point in the 3D-scene is projected. The displacement of the corresponding positions in the image planes is called disparity. The central problem in stereo vision, known as the correspondence problem, is to find corresponding points or features in the images. This task can be an ambiguous one due to several similar structures or periodic elements in the images. Furthermore, there may be occluded regions in the scene, which can be seen only by one camera. In these regions there is no solution for the correspondence problem. Interocular differences such as perspective distortions, differences in illumination and camera noise make it even more difficult to solve the correspondence problem. The main focus of this work is a new stereo matching algorithm, in which the matching of occluded areas is suppressed by a self-organizing process. In the first step the images are filtered by a set of oriented Gabor filters. A complex valued correlation-based similarity measurement, which is applied to the responses of the Gabor filters, is used in the second step to initialize a self-organizing process. In this self-organizing network, which is described by coupled, non-linear evolution equations, the continuity and the uniqueness constraints are established. Occlusions are detected implicitly without a computationally intensive bidirectional matching strategy.von Dipl.-Ing. Ralph Trapp aus Winterberg. Referent: Prof. Dr. rer. nat Georg Hartmann, Korreferent: Prof. Dr.-Ing. Ulrich RückertTag der Verteidigung: 15.09.1998Universität Paderborn, Univ., Dissertation, 199

Real-Time Algorithms for High Dynamic Range Video

A recurring problem in capturing video is the scene having a range of brightness values that exceeds the capabilities of the capturing device. An example would be a video camera in a bright outside area, directed at the entrance of a building. Because of the potentially big brightness difference, it may not be possible to capture details of the inside of the building and the outside simultaneously using just one shutter speed setting. This results in under- and overexposed pixels in the video footage. The approach we follow in this thesis to overcome this problem is temporal exposure bracketing, i.e., using a set of images captured in quick sequence at different shutter settings. Each image then captures one facet of the scene's brightness range. When fused together, a high dynamic range (HDR) video frame is created that reveals details in dark and bright regions simultaneously. The process of creating a frame in an HDR video can be thought of as a pipeline where the output of each step is the input to the subsequent one. It begins by capturing a set of regular images using varying shutter speeds. Next, the images are aligned with respect to each other to compensate for camera and scene motion during capture. The aligned images are then merged together to create a single HDR frame containing accurate brightness values of the entire scene. As a last step, the HDR frame is tone mapped in order to be displayable on a regular screen with a lower dynamic range. This thesis covers algorithms for these steps that allow the creation of HDR video in real-time. When creating videos instead of still images, the focus lies on high capturing and processing speed and on assuring temporal consistency between the video frames. In order to achieve this goal, we take advantage of the knowledge gained from the processing of previous frames in the video. This work addresses the following aspects in particular. The image size parameters for the set of base images are chosen such that only as little image data as possible is captured. We make use of the fact that it is not always necessary to capture full size images when only small portions of the scene require HDR. Avoiding redundancy in the image material is an obvious approach to reducing the overall time taken to generate a frame. With the aid of the previous frames, we calculate brightness statistics of the scene. The exposure values are chosen in a way, such that frequently occurring brightness values are well-exposed in at least one of the images in the sequence. The base images from which the HDR frame is created are captured in quick succession. The effects of intermediate camera motion are thus less intense than in the still image case, and a comparably simpler camera motion model can be used. At the same time, however, there is much less time available to estimate motion. For this reason, we use a fast heuristic that makes use of the motion information obtained in previous frames. It is robust to the large brightness difference between the images of an exposure sequence. The range of luminance values of an HDR frame must be tone mapped to the displayable range of the output device. Most available tone mapping operators are designed for still images and scale the dynamic range of each frame independently. In situations where the scene's brightness statistics change quickly, these operators produce visible image flicker. We have developed an algorithm that detects such situations in an HDR video. Based on this detection, a temporal stability criterion for the tone mapping parameters then prevents image flicker. All methods for capture, creation and display of HDR video introduced in this work have been fully implemented, tested and integrated into a running HDR video system. The algorithms were analyzed for parallelizability and, if applicable, adjusted and implemented on a high-performance graphics chip

Computing 3-D Motion in Custom Analog and Digital VLSI

This thesis examines a complete design framework for a real-time, autonomous system with specialized VLSI hardware for computing 3-D camera motion. In the proposed architecture, the first step is to determine point correspondences between two images. Two processors, a CCD array edge detector and a mixed analog/digital binary block correlator, are proposed for this task. The report is divided into three parts. Part I covers the algorithmic analysis; part II describes the design and test of a 32\time 32 CCD edge detector fabricated through MOSIS; and part III compares the design of the mixed analog/digital correlator to a fully digital implementation

Mise en correspondance stéréoscopique d'images couleur en présence d'occultations

This work deals with stereo-vision and more precisely matching of pixels using correlation measures. Matching is an important task in computer vision, the accuracy of the three-dimensional reconstruction depending on the accuracy of the matching. The problems of matching are: intensity distortions, noises, untextured areas, foreshortening and occlusions. Our research concerns matching color images and takes into account the problem of occlusions.First, we distinguish the different elements that can compose a matching algorithm. This description allows us to introduce a classification of matching methods into four families : local methods, global methods, mixed methods and multi-pass methods.Second, we set up an evaluation and comparison protocol based on fourteen image pairs, five evaluation areas and ten criteria. This protocol also provides disparity, ambiguity, inaccuracy and correct disparity maps. This protocol enables us to study the behavior of the methods we proposed.Third, forty correlation measures are classified into five families : cross-correlation-based measures, classical statistics-based measures, derivative-based measures, non-parametric measures and robust measures. We also propose six new measures based on robust statistics. The results show us the most robust measures near occlusions : the robust measures including the six new measures.Fourth, we propose to generalize dense correlation-based matching to color by choosing a color system and by generalizing the correlation measures to color. Ten color systems have been evaluated and three different methods have been compared : to compute the correlation with each color component and then to merge the results; to process a principal component analysis and then to compute the correlation with the first principal component; to compute the correlation directly with colors. We can conclude that the fusion method is the best.Finally, in order to take into account the problem of occlusions, we present new algorithms that use two correlation measures: a classic measure in non-occluded area and a robust measure in the whole occlusion area. We introduce four different methods: edge detection methods, weighted correlation methods, post-detection methods and fusion method. This latter method is the most efficient.Cette thèse se situe dans le cadre de la vision par ordinateur et concerne plus précisément l'étape de mise en correspondance de pixels en stéréovision binoculaire. Cette étape consiste à retrouver les pixels homologues dans deux images d'une même scène, prises de deux points de vue différents. Une des manières de réaliser la mise en correspondance est de faire appel à des mesures de corrélation. Les algorithmes utilisés se heurtent alors aux difficultés suivantes : les changements de luminosité, les bruits, les raccourcissements, les zones peu texturées et les occultations. Les travaux qui ont été réalisés sont une étude sur les méthodes à base de corrélation, en prenant en compte le problème des occultations et l'utilisation d'images couleur.Dans un premier chapitre, nous établissons un état de l'art des méthodes de mise en correspondance de pixels. Nous donnons un modèle générique des méthodes s'appuyant sur la définition d'éléments constituants. Nous distinguons alors quatre catégories de méthodes : les méthodes locales, les méthodes globales, les méthodes mixtes et les méthodes à multiples passages. Le second chapitre aborde le problème de l'évaluation des méthodes de mise en correspondance de pixels. Après avoir donné un état de l'art des protocoles existants, nous proposons un protocole d'évaluation et de comparaison qui prend en compte des images avec vérité terrain et qui distingue différentes zones d'occultations. Dans le troisième chapitre, nous proposons une taxonomie des mesures de corrélation regroupées en cinq familles : les mesures de corrélation croisée, les mesures utilisant des outils de statistiques classiques, les mesures utilisant les dérivées des images, les mesures s'appuyant sur des outils des statistiques non paramétriques et les mesures exploitant des outils des statistiques robustes. Parmi cette dernière famille, nous proposons dix-sept mesures. Les résultats obtenus avec notre protocole montrent que ces mesures obtiennent les meilleurs résultats dans les zones d'occultations. Le quatrième chapitre concerne la généralisation à la couleur des méthodes de mise en correspondance à base de corrélation. Après avoir présenté les systèmes de représentation de la couleur que nous testons, nous abordons la généralisation des méthodes à base de corrélation en passant par l'adaptation des mesures de corrélation à la couleur. Nous proposons trois méthodes différentes : fusion des résultats sur chaque composante, utilisation d'une analyse en composante principale et utilisation d'une mesure de corrélation couleur. Les résultats obtenus avec notre protocole mettent en évidence la meilleure méthode qui consiste à fusionner les scores de corrélation. Dans le dernier chapitre, pour prendre en compte les occultations, nous proposons des méthodes hybrides qui s'appuient sur l'utilisation de deux mesures de corrélation : une mesure classique dans les zones sans occultation et une mesure robuste dans les zones d'occultations. Nous distinguons quatre types de méthodes à base de détection de contours, de corrélation pondérée, de post-détection des occultations et de fusion de cartes de disparités. Les résultats obtenus avec notre protocole montrent que la méthode la plus performante consiste à fusionner deux cartes de disparités