Local Stereo Matching Using Adaptive Local Segmentation

We propose a new dense local stereo matching framework for gray-level images based on an adaptive local segmentation using a dynamic threshold. We define a new validity domain of the fronto-parallel assumption based on the local intensity variations in the 4-neighborhood of the matching pixel. The preprocessing step smoothes low textured areas and sharpens texture edges, whereas the postprocessing step detects and recovers occluded and unreliable disparities. The algorithm achieves high stereo reconstruction quality in regions with uniform intensities as well as in textured regions. The algorithm is robust against local radiometrical differences; and successfully recovers disparities around the objects edges, disparities of thin objects, and the disparities of the occluded region. Moreover, our algorithm intrinsically prevents errors caused by occlusion to propagate into nonoccluded regions. It has only a small number of parameters. The performance of our algorithm is evaluated on the Middlebury test bed stereo images. It ranks highly on the evaluation list outperforming many local and global stereo algorithms using color images. Among the local algorithms relying on the fronto-parallel assumption, our algorithm is the best ranked algorithm. We also demonstrate that our algorithm is working well on practical examples as for disparity estimation of a tomato seedling and a 3D reconstruction of a face

Occlusion-Aware Depth Estimation with Adaptive Normal Constraints

We present a new learning-based method for multi-frame depth estimation from a color video, which is a fundamental problem in scene understanding, robot navigation or handheld 3D reconstruction. While recent learning-based methods estimate depth at high accuracy, 3D point clouds exported from their depth maps often fail to preserve important geometric feature (e.g., corners, edges, planes) of man-made scenes. Widely-used pixel-wise depth errors do not specifically penalize inconsistency on these features. These inaccuracies are particularly severe when subsequent depth reconstructions are accumulated in an attempt to scan a full environment with man-made objects with this kind of features. Our depth estimation algorithm therefore introduces a Combined Normal Map (CNM) constraint, which is designed to better preserve high-curvature features and global planar regions. In order to further improve the depth estimation accuracy, we introduce a new occlusion-aware strategy that aggregates initial depth predictions from multiple adjacent views into one final depth map and one occlusion probability map for the current reference view. Our method outperforms the state-of-the-art in terms of depth estimation accuracy, and preserves essential geometric features of man-made indoor scenes much better than other algorithms.Comment: ECCV 202

A Nonlocal Method with Modified Initial Cost and Multiple Weight for Stereo Matching

This paper presents a new nonlocal cost aggregation method for stereo matching. The minimum spanning tree (MST) employs color difference as the sole component to build the weight function, which often leads to failure in achieving satisfactory results in some boundary regions with similar color distributions. In this paper, a modified initial cost is used. The erroneous pixels are often caused by two pixels from object and background, which have similar color distribution. And then inner color correlation is employed as a new component of the weight function, which is determined to effectively eliminate them. Besides, the segmentation method of the tree structure is also improved. Thus, a more robust and reasonable tree structure is developed. The proposed method was tested on Middlebury datasets. As can be expected, experimental results show that the proposed method outperforms the classical nonlocal methods

Multi Cost Function Fuzzy Stereo Matching Algorithm for Object Detection and Robot Motion Control

Stereo matching algorithms work with multiple images of a scene, taken from two viewpoints, to generate depth information. Authors usually use a single matching function to generate similarity between corresponding regions in the images. In the present research, the authors have considered a combination of multiple data costs for disparity generation. Disparity maps generated from stereo images tend to have noisy sections. The presented research work is related to a methodology to refine such disparity maps such that they can be further processed to detect obstacle regions.  A novel entropy based selective refinement (ESR) technique is proposed to refine the initial disparity map. The information from both the left disparity and right disparity maps are used for this refinement technique. For every disparity map, block wise entropy is calculated. The average entropy values of the corresponding positions in the disparity maps are compared. If the variation between these entropy values exceeds a threshold, then the corresponding disparity value is replaced with the mean disparity of the block with lower entropy. The results of this refinement are compared with similar methods and was observed to be better. Furthermore, in this research work, the v-disparity values are used to highlight the road surface in the disparity map. The regions belonging to the sky are removed through HSV based segmentation. The remaining regions which are our ROIs, are refined through a u-disparity area-based technique.  Based on this, the closest obstacles are detected through the use of k-means segmentation.  The segmented regions are further refined through a u-disparity image information-based technique and used as masks to highlight obstacle regions in the disparity maps. This information is used in conjunction with a kalman filter based path planning algorithm to guide a mobile robot from a source location to a destination location while also avoiding any obstacle detected in its path. A stereo camera setup was built and the performance of the algorithm on local real-life images, captured through the cameras, was observed. The evaluation of the proposed methodologies was carried out using real life out door images obtained from KITTI dataset and images with radiometric variations from Middlebury stereo dataset

Field-based Robot Phenotyping of Sorghum Plant Architecture using Stereo Vision

Sorghum (Sorghum bicolor) is known as a major feedstock for biofuel production. To improve its biomass yield through genetic research, manually measuring yield component traits (e.g. plant height, stem diameter, leaf angle, leaf area, leaf number, and panicle size) in the field is the current best practice. However, such laborious and time‐consuming tasks have become a bottleneck limiting experiment scale and data acquisition frequency. This paper presents a high‐throughput field‐based robotic phenotyping system which performed side‐view stereo imaging for dense sorghum plants with a wide range of plant heights throughout the growing season. Our study demonstrated the suitability of stereo vision for field‐based three‐dimensional plant phenotyping when recent advances in stereo matching algorithms were incorporated. A robust data processing pipeline was developed to quantify the variations or morphological traits in plant architecture, which included plot‐based plant height, plot‐based plant width, convex hull volume, plant surface area, and stem diameter (semiautomated). These image‐derived measurements were highly repeatable and showed high correlations with the in‐field manual measurements. Meanwhile, manually collecting the same traits required a large amount of manpower and time compared to the robotic system. The results demonstrated that the proposed system could be a promising tool for large‐scale field‐based high‐throughput plant phenotyping of bioenergy crops

Optimizing 3D Convolutions on Stereo Matching for Resource Efficient Computation

13301甲第5510号博士(工学)金沢大学博士論文本文Full 以下に掲載：Sensors 21(20) pp.6808 2021. MDPI. 共著者：Jianqiang Xiao, Dianbo Ma, Satoshi Yaman

Метод фотограмметрії дорожньої обстановки

Дисертацію виконано на 84 аркушах, вона містить 2 додатки та перелік посилань на використані джерела з 41 найменувань. У роботі наведено 42 рисунки та 20 таблиц. Актуальність теми. На сьогоднішній день у світі спостерігається суттєве зростання інтересу до створення штучного інтелекту. Вже планується і вже частково впроваджено системи, які керують автомобілем без участі водія. Основним елементом даних систем є аналіз дорожньої обстановки, який, в основному, проводиться за допомогою пари стерео камер. Для вивчення положень об’єктів, їх параметрів використовується методи фотограмметрії, а саме стереофотограмметрія. За допомогою неї можна оцінювати відстань до об’єктів навколишнього середовища, дізнатися координати об’єкта в трьох вимірному просторі. Саме знаючи відстань до об’єктів, що оточують автомобіль, можна правильно прийняти рішення про подальший рух автомобіля, правильно здійснювати керування транспортним засобом, можливість прогнозувати подальший рух об’єктів та корегувати власну траєкторію руху. Зв’язок роботи з науковими програмами, планами, темами. Дисертаційна робота виконувалась згідно з планом науково-дослідних робіт кафедри Математичних методів системного аналізу Національного технічного університету України «Київський політехнічний інститут імені Ігоря Сікорського». Мета і задачі дослідження. Метою дисертаційної роботи є розробка методу оцінки положення об’єктів дорожньої обстановки за допомогою методів фотограмметрії. Для досягнення вказаної мети було розв’язано такі задачі: провести аналіз методу фотограмметрії; провести аналіз основних проблем обробки фотографій для визначення невідповідності; розробити метод для оцінки глибини дорожньої обстановки; розробити систему для побудови карт глибини дорожньої обстановки зі стерео пар. Об’єктом дослідження є дорожня обстановка. Предметом дослідження є метод фотограмметрії. Методи дослідження. Для розв’язання задачі використовувалися методи стереофотограмметрії (для розробки моделі побудови карти відстаней); методи оптимізації (для надання необхідної точності обраної моделі); методи теорії алгоритмів та програмування (для програмної реалізації розроблених алгоритмів); методи теорії ймовірності та математичної статистики (для проведення оцінки якості запропонованої моделі). Наукова новизна одержаних результатів. Удосконалено модель дорожньої обстановки, яка, на відміну від існуючих, більш точно підраховує відстань до оточуючих об’єктів дорожньої обстановки. Практичне значення одержаних результатів. Запропоновано метод, який може бути використано як помічник водію під час руху автомобіля або для руху автомобіля без участі водія. Розроблений метод, математичне та програмне забезпечення дозволяють швидко та якісно отримувати карту глибини дорожньої обстановки. Публікації. Результати дисертації викладено в у міжнародному науковому журналі.The thesis is presented in 84 pages. It contains 2 appendixes and bibliography of 41 references. 42 figures and 20 tables are given in the thesis. Topic relevance. To date, in the world there is a significant increase in interest in the creation of artificial intelligence. Already planned and already partially introduced systems that control the car without the participation of the driver. The main element of these systems is the analysis of the road environment, which is mainly carried out using a pair of stereo cameras. To study the positions of objects, their parameters are used methods of photogrammetry, namely stereophotogrammeter. With it, you can estimate the distance to the objects of the environment, to find out the coordinates of the object in three-dimensional space. Just knowing the distance to the objects surrounding the car, you can correctly decide on the further movement of the car, correctly manage the vehicle, the ability to predict the further movement of objects and adjust their own trajectory of motion. Thesis connection to scientific programs, plans, and topics. The thesis was prepared according to the scientific research plan of the Mathematical Methods of System Analysis Department of the National Technical University of Ukraine “Igor Sikorsky Kyiv Polytechnic Institute.” Research goal and objectives. The goal of this thesis is to develop a method for evaluation the position of objects of the road with photogrammetry methods. To accomplish this goal, the following objectives were reached: analyze photogrammetry method; analyze main problems of the processing of photographs for disparity calculation; develop a method for depth calculation of the road environment; develop a system for depth map generation of the road environment utilizing stereo pairs. Object of research is road environment. Subject of research is photogrammetric method. Methods of research. To solve the task, the following methods were used: methods for stereophotogrammetry (for developing a model for mapping distances); methods of optimization (to provide the necessary accuracy of the chosen model); methods of algorithm theory and programming (for software implementation of developed algorithms); methods of probability theory and mathematical statistics (for evaluation of the quality of the proposed model). Scientific contribution consists of the following: the model of the road environment, which, in contrast to the existing, more accurately calculates the distance to surrounding objects. Practical value of obtained results. The proposed method can be used as a driver assistant or for self-driving cars. The developed method and software allow you to quickly and precisely receive a depth map of the road environment. Publications. The results of the dissertation are presented in the international scientific journal