FPGA implementation of an efficient similarity-based adaptive window algorithm for real-time stereo matching

International audienceThe stereo matching is one of the most widely used algorithms in real-time image processing applications such as positioning systems for mobile robots, three-dimensional building mapping and both recognition, detection and three-dimensional reconstruction of objects. In area-based algorithms, the similarity between one pixel of the left image and one pixel of the right image is measured using a correlation index computed on vicinities of these pixels called correlation windows. In order to preserve edges, small windows need to be used. On the other hand, for homogeneous areas, large windows are required. Due to only local information is used, matching between primitives is difficult. In this article, FPGA implementing of an efficient similarity-based adaptive window algorithm for dense disparity maps estimation in real-time is described. In order to evaluate the proposed algorithm behavior, the developed FPGA architecture was simulated via ModelSim-Altera 6.6c using different synthetic stereo pairs and different sizes for correlation window. In addition, the FPGA architecture was implemented in an FPGA Cyclone IIEP2C35F672C6 embedded in an Altera development board DE2. The disparity maps are computed at a rate of 76 frames per second for stereo pairs of 1280×1024 pixel resolution and a maximum expected disparity equal to 15. The developed FPGA architecture offers better results with respect to the most of real-time area-based stereo matching algorithms reported in the literature, allows increasing the processing speed up to 93,061,120 pixels per second and enables it to be implemented in the majority of the medium-gamma FPGA devices

An FPGA-CAPH Stereo Matching Processor Based on the Sum of Hamming Distances

International audienceStereo matching is a useful algorithm to infer depth information from two or more of images and has uses in mobile robotics, three-dimensional building mapping and three-dimensional reconstruction of objects. In area-based algorithms, the similarity between one pixel of an image (key frame) and one pixel of another image is measured using a correlation index computed on neighbors of these pixels (correlation windows). In order to preserve edges, the use of small correlation windows is necessary while for homogeneous areas, large windows are required. In addition, to improve the execution time, stereo matching algorithms often are implemented in dedicated hardware such as FPGA or GPU devices. In this article, we present an FPGA stereo matching processor based on the Sum of Hamming Distances (SHD). We propose a grayscale-based similarity criterion, which allows separating the objects and background from the correlation window. By using the similarity criterion, it is possible to improve the performance of any grayscale-based correlation coefficient and reach high performance for homogeneous areas and edges. The developed FPGA architecture reaches high performance compared to other real-time stereo matching algorithms, up to 10 % more accuracy and enables to increase the processing speed near to 20 megapixels per second

Dense mapping for monocular-SLAM

International audienceSimultaneous Localization and Mapping (SLAM) is the problem of constructing a 3D map while simultaneously keeping track of an agent location within the map. In recent years, work has focused in systems that use a single camera as the only sensing mechanism (monocular-SLAM). 3D reconstruction (map) by monocular-SLAM systems is a point cloud where all points preserve high accuracy and can deliver visual environmental information. However, the maximum number of points in the cloud is limited by the tracked features, this is named " sparse cloud problem ". In this work, we propose a new SLAM framework that is robust enough for indoor/outdoor SLAM applications, and at the same time increases the 3D map density. The point cloud density is increased by applying a new feature-tracking/dense-tracking algorithm in the SLAM formulation. In order to achieve real-time processing, the algorithm is formulated to facilitate a parallel FPGA implementation. Preliminary results show that it is possible to obtain dense mapping (superior to previous work) and accurate camera pose estimation (localization) under several real-world conditions

Depth from motion algorithm and hardware architecture for smart cameras

International audienceApplications such as autonomous navigation, robot vision, autonomous flying, etc., require 1 depth map information of the scene. Depth can be estimated by using a single moving camera 2 (depth from motion). However, traditional depth from motion algorithms have low processing speed 3 and high hardware requirements that limits the embedded capabilities. In this work, we propose 4 a hardware architecture for depth from motion that consists of a flow/depth transformation and 5 a new optical flow algorithm. Our optical flow formulation consists in an extension of the stereo 6 matching problem. A pixel-parallel/window-parallel approach where a correlation function based in 7 the Sum of Absolute Differences computes the optical flow is proposed. Further, in order to improve 8 the Sum of Absolute Differences performance, the curl of the intensity gradient as preprocessing step 9 is proposed. Experimental results demonstrated that it is possible to reach higher accuracy (90% of 10 accuracy) compared with previous FPGA-based optical flow algorithms. For the depth estimation, 11 our algorithm delivers dense maps with motion and depth information on all the image pixels, with 12 a processing speed up to 128 times faster than previous works and making it possible to achieve high 13 performance in the context of embedded applications. 1

Robust feature extraction algorithm suitable for real-time embedded applications

International audienceSmart cameras integrate processing close to the image sensor, so they can deliver high-level information to a host computer or high-level decision process. One of the most common processing is the visual features extraction since many vision-based use-cases are based on such algorithm. Unfortunately, in most of cases, features detection algorithms are not robust or do not reach real-time processing. Based on these limitations, a feature detection algorithm that is robust enough to deliver robust features under any type of indoor / outdoor scenarios is proposed. This was achieved by applying a non-textured corner filter combined to a subpixel refinement. Furthermore, an FPGA architecture is proposed. This architecture allows compact system design, real-time processing for Full HD images (it can process up to 44 frames/91.238.400 pixels per second for Full HD images), and high efficiency for smart camera implementations (similar hardware resources than previous formulations without subpixel refinement and without non-textured corner filter). For accuracy/robustness, experimental results for several real world scenes are encouraging and show the feasibility of our algorithmic approach

The Fastest Visual Ego-motion Algorithm in the West

International audienc

PhD Forum: Camera Pose Estimation Suitable for Smart Cameras

International audienceCamera pose estimation across video sequences is an important issue under several computer vision applications. In previous work, the most popular approach consists on optimization techniques applied over 2D/3D point correspondences for two consecutive frames from a video sequence. Unfortunately, these optimization techniques are iterative and depend on nonlinear optimizations applied over some geometric constraint. For real-time embedded applications, another approach, more efficient in terms of computational size and cost, could be a linear or closed-form solution for the camera pose estimation problem. In this work, we introduce a new approach for camera pose estimation, this approach uses 2D visual features displacements as linear/dependent parameters for the camera pose estimation so, camera pose can be estimated without iterative behavior and without geometric constraints. As result, the proposed algorithm could be implemented inside a small FPGA device, suitable for smart cameras. Preliminary results are encourageous and show the viability of the proposed approach

Depth from a Motion Algorithm and a Hardware Architecture for Smart Cameras

Applications such as autonomous navigation, robot vision, and autonomous flying require depth map information of a scene. Depth can be estimated by using a single moving camera (depth from motion). However, the traditional depth from motion algorithms have low processing speeds and high hardware requirements that limit the embedded capabilities. In this work, we propose a hardware architecture for depth from motion that consists of a flow/depth transformation and a new optical flow algorithm. Our optical flow formulation consists in an extension of the stereo matching problem. A pixel-parallel/window-parallel approach where a correlation function based on the sum of absolute difference (SAD) computes the optical flow is proposed. Further, in order to improve the SAD, the curl of the intensity gradient as a preprocessing step is proposed. Experimental results demonstrated that it is possible to reach higher accuracy (90% of accuracy) compared with previous Field Programmable Gate Array (FPGA)-based optical flow algorithms. For the depth estimation, our algorithm delivers dense maps with motion and depth information on all image pixels, with a processing speed up to 128 times faster than that of previous work, making it possible to achieve high performance in the context of embedded applications

PhD Forum: GPU-based Visual Odometry for Autonomous Vehicle Applications

International audienceOne popular task under several computer vision applications is camera pose estimation under video sequences. In previous work, several camera pose estimations approaches have been developed and several algorithms have been proposed. Unfortunately, most previous formulations iterative behavior and depend on nonlinear optimizations applied over some geometric constraint and this limits the performance under embedded applications. For real-time embedded applications , another approach, more efficient in terms of computational size and processing speed could be reached via hardware-based solutions, for example GPU-based solutions. In this work, we present a GPU-based solution for the camera pose problem. As early formulation we focused our algorithm for an autonomous vehicle application. Preliminary results are encouraging and show the viability of the proposed approach

Dense Feature Matching Core for FPGA-based Smart Cameras

International audienceSmart cameras are image/video acquisition devices that integrate image processing algorithms close to the image sensor, so they can deliver high-level information to a host computer or high-level decision process. In this context, a central issue is the implementation of complex and computationally intensive computer vision algorithms inside the camera fabric. For low-level processing, FPGA devices are excellent candidates because they support data paral-lelism with high data throughput. One computer vision algorithm highly promising for FPGA-based smart cameras is feature matching. Unfortunately, most previous feature matching formulations have inefficient FPGA implementations or deliver relatively poor information about the observed scene. In this work, we introduce a new feature-matching algorithm that aims for dense feature matching and at the same time straightforward FPGA implementation. We propose a new mathematical formulation that addressed the feature matching task as a feature tracking problem. We demonstrate that our algorithmic formulation delivers robust feature matching with low mathematical complexity and obtains accuracy superior to previous algorithmic formulations. An FPGA architecture is lay down and, hardware acceleration strategies are discussed. Finally , we applied our feature matching algorithm in a monocular-SLAM system. We show that our algorithmic formulation provides promising results under real world applications