Real time architectures for the scale Invariant feature transform algorithm

Feature extraction in digital image processing is a very intensive task for a CPU. In order to achieve real time image throughputs, hardware parallelism must be exploited. The speed-up of the system is constrained by the degree of parallelism of the implementation and this one at the same time, by programmable device size and the power dissipation. In this work, issues related to the synthesis of the Scale-Invariant Feature Transform (SIFT) algorithm on a FPGA to obtain target processing rates faster than 50 frames per second for VGA images, are analyzed. In order to increase the speedup of the algorithm, the work includes the analysis of feasible simplifications of the algorithm for a tracking application and the results are synthesized on an FPGA.This work has been partially funded by Spanish government projects TEC2015-66878-C3-2-R (MINECO/FEDER, UE) and TEC2015- 66878-C3-3-R (MINECO/FEDER, UE)

FPGA synthesis of an stereo image matching architecture for autonomous mobile robots

This paper describes a hardware proposal to speed up the process of image matching in stereo vision systems like those employed by autonomous mobile robots. This proposal combines a classical window-based matching approach with a previous stage, where key points are selected from each image of the stereo pair. In this first step the key point extraction method is based on the SIFT algorithm. Thus, in the second step, the window-based matching is only applied to the set of selected key points, instead of to the whole images. For images with a 1% of key points, this method speeds up the matching four orders of magnitude. This proposal is, on the one hand, a better parallelizable architecture than the original SIFT, and on the other, a faster technique than a full image windows matching approach. The architecture has been implemented on a lower power Virtex 6 FPGA and it achieves a image matching speed above 30 fps.This work has been funded by Spanish government project TEC2015-66878-C3-2-R (MINECO/FEDER, UE)

A high-performance hardware architecture of an image matching system based on the optimised SIFT algorithm

The Scale Invariant Feature Transform (SIFT) is one of the most popular matching algorithms in the field of computer vision. It takes over many other algorithms because features detected are fully invariant to image scaling and rotation, and are also shown to be robust to changes in 3D viewpoint, addition of noise, changes in illumination and a sustainable range of affine distortion. However, the computational complexity is high, which prevents it from achieving real-time. The aim of this project, therefore, is to develop a high-performance image matching system based on the optimised SIFT algorithm to perform real-time feature detection, description and matching. This thesis presents the stages of the development of the system. To reduce the computational complexity, an alternative to the grid layout of standard SIFT is proposed, which is termed as SRI-DASIY (Scale and Rotation Invariant DAISY). The SRI-DAISY achieves comparable performance with the standard SIFT descriptor, but is more efficient to be implemented using hardware, in terms of both computational complexity and memory usage. The design takes only 7.57 µs to generate a descriptor with a system frequency of 100 MHz, which is equivalent to approximately 132,100 descriptors per second and is of the highest throughput when compared with existing designs. Besides, a novel keypoint matching strategy is also presented in this thesis, which achieves higher precision than the widely applied distance ratio based matching and is computationally more efficient. All phases of the SIFT algorithm have been investigated, including feature detection, descriptor generation and descriptor matching. The characterisation of each individual part of the design is carried out and compared with the software simulation results. A fully stand-alone image matching system has been developed that consists of a CMOS camera front-end for image capture, a SIFT processing core embedded in a Field Programmable Logic Array (FPGA) device, and a USB back-end for data transfer. Experiments are conducted by using real-world images to verify the system performance. The system has been tested by integrating into two practical applications. The resulting image matching system eliminates the bottlenecks that limit the overall throughput of the system, and hence allowing the system to process images in real-time without interruption. The design can be modified to adapt to the applications processing images with higher resolution and is still able to achieve real-time

FPGA-based module for SURF extraction

We present a complete hardware and software solution of an FPGA-based computer vision embedded module capable of carrying out SURF image features extraction algorithm. Aside from image analysis, the module embeds a Linux distribution that allows to run programs specifically tailored for particular applications. The module is based on a Virtex-5 FXT FPGA which features powerful configurable logic and an embedded PowerPC processor. We describe the module hardware as well as the custom FPGA image processing cores that implement the algorithm's most computationally expensive process, the interest point detection. The module's overall performance is evaluated and compared to CPU and GPU based solutions. Results show that the embedded module achieves comparable disctinctiveness to the SURF software implementation running in a standard CPU while being faster and consuming significantly less power and space. Thus, it allows to use the SURF algorithm in applications with power and spatial constraints, such as autonomous navigation of small mobile robots

A high-performance hardware architecture of an image matching system based on the optimised SIFT algorithm

The Scale Invariant Feature Transform (SIFT) is one of the most popular matching algorithms in the field of computer vision. It takes over many other algorithms because features detected are fully invariant to image scaling and rotation, and are also shown to be robust to changes in 3D viewpoint, addition of noise, changes in illumination and a sustainable range of affine distortion. However, the computational complexity is high, which prevents it from achieving real-time. The aim of this project, therefore, is to develop a high-performance image matching system based on the optimised SIFT algorithm to perform real-time feature detection, description and matching. This thesis presents the stages of the development of the system. To reduce the computational complexity, an alternative to the grid layout of standard SIFT is proposed, which is termed as SRI-DASIY (Scale and Rotation Invariant DAISY). The SRI-DAISY achieves comparable performance with the standard SIFT descriptor, but is more efficient to be implemented using hardware, in terms of both computational complexity and memory usage. The design takes only 7.57 µs to generate a descriptor with a system frequency of 100 MHz, which is equivalent to approximately 132,100 descriptors per second and is of the highest throughput when compared with existing designs. Besides, a novel keypoint matching strategy is also presented in this thesis, which achieves higher precision than the widely applied distance ratio based matching and is computationally more efficient. All phases of the SIFT algorithm have been investigated, including feature detection, descriptor generation and descriptor matching. The characterisation of each individual part of the design is carried out and compared with the software simulation results. A fully stand-alone image matching system has been developed that consists of a CMOS camera front-end for image capture, a SIFT processing core embedded in a Field Programmable Logic Array (FPGA) device, and a USB back-end for data transfer. Experiments are conducted by using real-world images to verify the system performance. The system has been tested by integrating into two practical applications. The resulting image matching system eliminates the bottlenecks that limit the overall throughput of the system, and hence allowing the system to process images in real-time without interruption. The design can be modified to adapt to the applications processing images with higher resolution and is still able to achieve real-time

Towards a Common Software/Hardware Methodology for Future Advanced Driver Assistance Systems

The European research project DESERVE (DEvelopment platform for Safe and Efficient dRiVE, 2012-2015) had the aim of designing and developing a platform tool to cope with the continuously increasing complexity and the simultaneous need to reduce cost for future embedded Advanced Driver Assistance Systems (ADAS). For this purpose, the DESERVE platform profits from cross-domain software reuse, standardization of automotive software component interfaces, and easy but safety-compliant integration of heterogeneous modules. This enables the development of a new generation of ADAS applications, which challengingly combine different functions, sensors, actuators, hardware platforms, and Human Machine Interfaces (HMI). This book presents the different results of the DESERVE project concerning the ADAS development platform, test case functions, and validation and evaluation of different approaches. The reader is invited to substantiate the content of this book with the deliverables published during the DESERVE project. Technical topics discussed in this book include:Modern ADAS development platforms;Design space exploration;Driving modelling;Video-based and Radar-based ADAS functions;HMI for ADAS;Vehicle-hardware-in-the-loop validation system

An investigation into combining both facial detection and landmark localisation into a unified procedure using GPU computing

This thesis describes the design and implementation of a unified framework for face detection and landmark alignment in arbitrary in the wild images. Traditionally, both of these problems have been addressed separately in literature with impressive results being recently reported in both of these fields. But, if one was to construct a pipeline consisting of a state-of-the-art face detection method followed by a state-of-the-art facial landmark localisation algorithm, the overall performance outcome would not be proficient enough to be used in high level algorithms such as face recognition and facial expression. This is because the accuracy produced by the face detector is not sufficiently high enough to initialise the landmark localisation algorithm. To address this aforementioned limitation, this thesis aims to propose an approach that combines both of these tasks into a single unified algorithm that can be run in real time, by utilising the parallel computing architecture of the graphics processing unit (GPU). This will be done by using a Cascaded-Regression (CR) algorithm in a sliding window fashion. The proposed system will exploit the CR algorithms ability to compute the 2D pose of a face from rough initial estimates, in order to generate a Hough- Transform voting scheme for detecting candidate faces and filtering out irrelevant background. The obtained detection surface will then be further refined using SVM to yield both face detections and the location of their parts. The proposed system for this thesis will be built within the MATLAB environment, using a MEX-file which will provide an interface to the proposed CUDA algorithm. The results of which, will be tested against current state-of-the-art methods for both face detection and landmark localisation. We evaluate performance on the most widely used data sets in face detection, namely annotated faces in-the-wild (AFW) (Zhu and Ramanan, 2012), Face Detection Dataset and Benchmark (FDDB) (Jain and Learned-Miller, 2010) and Caltech Occluded Faces in the Wild (COFW) (Burgos-Artizzu, Perona and Dollár, 2013). The empirical results demonstrate that the proposed unified framework achieves state-of-the-art performance in both face detection and facial alignment, and that our detector clearly outperforms all commercial and published methods by a margin of over 10% in detection accuracy on the AFW dataset