Hardware Accelarated Visual Tracking Algorithms. A Systematic Literature Review

Many industrial applications need object recognition and tracking capabilities. The algorithms developed for those purposes are computationally expensive. Yet ,real time performance, high accuracy and small power consumption are essential measures of the system. When all these requirements are combined, hardware acceleration of these algorithms becomes a feasible solution. The purpose of this study is to analyze the current state of these hardware acceleration solutions, which algorithms have been implemented in hardware and what modifications have been done in order to adapt these algorithms to hardware.Siirretty Doriast

ENERGY-EFFICIENT FEATURE EXTRACTION ENGINE AND SECURE CHIP IDENTIFICATION FOR UBIQUITOUS SURVEILLANCE

Ph.DDOCTOR OF PHILOSOPH

Cable Tension Monitoring using Non-Contact Vision-based Techniques

In cable-stayed bridges, the structural systems of tensioned cables play a critical role in structural and functional integrity. Thereby, tensile forces in the cables become one of the essential indicators in structural health monitoring (SHM). In this thesis, a video image processing technology integrated with cable dynamic analysis is proposed as a non-contact vision-based measurement technique, which provides a user-friendly, cost-effective, and computationally efficient solution to displacement extraction, frequency identification, and cable tension monitoring. In contrast to conventional contact sensors, the vision-based system is capable of taking remote measurements of cable dynamic response while having flexible sensing capability. Since cable detection is a substantial step in displacement extraction, a comprehensive study on the feasibility of the adopted feature detector is conducted under various testing scenarios. The performance of the feature detector is quantified by developing evaluation parameters. Enhancement methods for the feature detector in cable detection are investigated as well under complex testing environments. Threshold-dependent image matching approaches, which optimize the functionality of the feature-based video image processing technology, is proposed for noise-free and noisy background scenarios. The vision-based system is validated through experimental studies of free vibration tests on a single undamped cable in laboratory settings. The maximum percentage difference of the identified cable fundamental frequency is found to be 0.74% compared with accelerometer readings, while the maximum percentage difference of the estimated cable tensile force is 4.64% compared to direct measurement by a load cell

High-performance hardware accelerators for image processing in space applications

Mars is a hard place to reach. While there have been many notable success stories in getting probes to the Red Planet, the historical record is full of bad news. The success rate for actually landing on the Martian surface is even worse, roughly 30%. This low success rate must be mainly credited to the Mars environment characteristics. In the Mars atmosphere strong winds frequently breath. This phenomena usually modifies the lander descending trajectory diverging it from the target one. Moreover, the Mars surface is not the best place where performing a safe land. It is pitched by many and close craters and huge stones, and characterized by huge mountains and hills (e.g., Olympus Mons is 648 km in diameter and 27 km tall). For these reasons a mission failure due to a landing in huge craters, on big stones or on part of the surface characterized by a high slope is highly probable. In the last years, all space agencies have increased their research efforts in order to enhance the success rate of Mars missions. In particular, the two hottest research topics are: the active debris removal and the guided landing on Mars. The former aims at finding new methods to remove space debris exploiting unmanned spacecrafts. These must be able to autonomously: detect a debris, analyses it, in order to extract its characteristics in terms of weight, speed and dimension, and, eventually, rendezvous with it. In order to perform these tasks, the spacecraft must have high vision capabilities. In other words, it must be able to take pictures and process them with very complex image processing algorithms in order to detect, track and analyse the debris. The latter aims at increasing the landing point precision (i.e., landing ellipse) on Mars. Future space-missions will increasingly adopt Video Based Navigation systems to assist the entry, descent and landing (EDL) phase of space modules (e.g., spacecrafts), enhancing the precision of automatic EDL navigation systems. For instance, recent space exploration missions, e.g., Spirity, Oppurtunity, and Curiosity, made use of an EDL procedure aiming at following a fixed and precomputed descending trajectory to reach a precise landing point. This approach guarantees a maximum landing point precision of 20 km. By comparing this data with the Mars environment characteristics, it is possible to understand how the mission failure probability still remains really high. A very challenging problem is to design an autonomous-guided EDL system able to even more reduce the landing ellipse, guaranteeing to avoid the landing in dangerous area of Mars surface (e.g., huge craters or big stones) that could lead to the mission failure. The autonomous behaviour of the system is mandatory since a manual driven approach is not feasible due to the distance between Earth and Mars. Since this distance varies from 56 to 100 million of km approximately due to the orbit eccentricity, even if a signal transmission at the light speed could be possible, in the best case the transmission time would be around 31 minutes, exceeding so the overall duration of the EDL phase. In both applications, algorithms must guarantee self-adaptability to the environmental conditions. Since the Mars (and in general the space) harsh conditions are difficult to be predicted at design time, these algorithms must be able to automatically tune the internal parameters depending on the current conditions. Moreover, real-time performances are another key factor. Since a software implementation of these computational intensive tasks cannot reach the required performances, these algorithms must be accelerated via hardware. For this reasons, this thesis presents my research work done on advanced image processing algorithms for space applications and the associated hardware accelerators. My research activity has been focused on both the algorithm and their hardware implementations. Concerning the first aspect, I mainly focused my research effort to integrate self-adaptability features in the existing algorithms. While concerning the second, I studied and validated a methodology to efficiently develop, verify and validate hardware components aimed at accelerating video-based applications. This approach allowed me to develop and test high performance hardware accelerators that strongly overcome the performances of the actual state-of-the-art implementations. The thesis is organized in four main chapters. Chapter 2 starts with a brief introduction about the story of digital image processing. The main content of this chapter is the description of space missions in which digital image processing has a key role. A major effort has been spent on the missions in which my research activity has a substantial impact. In particular, for these missions, this chapter deeply analizes and evaluates the state-of-the-art approaches and algorithms. Chapter 3 analyzes and compares the two technologies used to implement high performances hardware accelerators, i.e., Application Specific Integrated Circuits (ASICs) and Field Programmable Gate Arrays (FPGAs). Thanks to this information the reader may understand the main reasons behind the decision of space agencies to exploit FPGAs instead of ASICs for high-performance hardware accelerators in space missions, even if FPGAs are more sensible to Single Event Upsets (i.e., transient error induced on hardware component by alpha particles and solar radiation in space). Moreover, this chapter deeply describes the three available space-grade FPGA technologies (i.e., One-time Programmable, Flash-based, and SRAM-based), and the main fault-mitigation techniques against SEUs that are mandatory for employing space-grade FPGAs in actual missions. Chapter 4 describes one of the main contribution of my research work: a library of high-performance hardware accelerators for image processing in space applications. The basic idea behind this library is to offer to designers a set of validated hardware components able to strongly speed up the basic image processing operations commonly used in an image processing chain. In other words, these components can be directly used as elementary building blocks to easily create a complex image processing system, without wasting time in the debug and validation phase. This library groups the proposed hardware accelerators in IP-core families. The components contained in a same family share the same provided functionality and input/output interface. This harmonization in the I/O interface enables to substitute, inside a complex image processing system, components of the same family without requiring modifications to the system communication infrastructure. In addition to the analysis of the internal architecture of the proposed components, another important aspect of this chapter is the methodology used to develop, verify and validate the proposed high performance image processing hardware accelerators. This methodology involves the usage of different programming and hardware description languages in order to support the designer from the algorithm modelling up to the hardware implementation and validation. Chapter 5 presents the proposed complex image processing systems. In particular, it exploits a set of actual case studies, associated with the most recent space agency needs, to show how the hardware accelerator components can be assembled to build a complex image processing system. In addition to the hardware accelerators contained in the library, the described complex system embeds innovative ad-hoc hardware components and software routines able to provide high performance and self-adaptable image processing functionalities. To prove the benefits of the proposed methodology, each case study is concluded with a comparison with the current state-of-the-art implementations, highlighting the benefits in terms of performances and self-adaptability to the environmental conditions

Towards visualization and searching :a dual-purpose video coding approach

In modern video applications, the role of the decoded video is much more than filling a screen for visualization. To offer powerful video-enabled applications, it is increasingly critical not only to visualize the decoded video but also to provide efficient searching capabilities for similar content. Video surveillance and personal communication applications are critical examples of these dual visualization and searching requirements. However, current video coding solutions are strongly biased towards the visualization needs. In this context, the goal of this work is to propose a dual-purpose video coding solution targeting both visualization and searching needs by adopting a hybrid coding framework where the usual pixel-based coding approach is combined with a novel feature-based coding approach. In this novel dual-purpose video coding solution, some frames are coded using a set of keypoint matches, which not only allow decoding for visualization, but also provide the decoder valuable feature-related information, extracted at the encoder from the original frames, instrumental for efficient searching. The proposed solution is based on a flexible joint Lagrangian optimization framework where pixel-based and feature-based processing are combined to find the most appropriate trade-off between the visualization and searching performances. Extensive experimental results for the assessment of the proposed dual-purpose video coding solution under meaningful test conditions are presented. The results show the flexibility of the proposed coding solution to achieve different optimization trade-offs, notably competitive performance regarding the state-of-the-art HEVC standard both in terms of visualization and searching performance.Em modernas aplicações de vídeo, o papel do vídeo decodificado é muito mais que simplesmente preencher uma tela para visualização. Para oferecer aplicações mais poderosas por meio de sinais de vídeo,é cada vez mais crítico não apenas considerar a qualidade do conteúdo objetivando sua visualização, mas também possibilitar meios de realizar busca por conteúdos semelhantes. Requisitos de visualização e de busca são considerados, por exemplo, em modernas aplicações de vídeo vigilância e comunicações pessoais. No entanto, as atuais soluções de codificação de vídeo são fortemente voltadas aos requisitos de visualização. Nesse contexto, o objetivo deste trabalho é propor uma solução de codificação de vídeo de propósito duplo, objetivando tanto requisitos de visualização quanto de busca. Para isso, é proposto um arcabouço de codificação em que a abordagem usual de codificação de pixels é combinada com uma nova abordagem de codificação baseada em features visuais. Nessa solução, alguns quadros são codificados usando um conjunto de pares de keypoints casados, possibilitando não apenas visualização, mas também provendo ao decodificador valiosas informações de features visuais, extraídas no codificador a partir do conteúdo original, que são instrumentais em aplicações de busca. A solução proposta emprega um esquema flexível de otimização Lagrangiana onde o processamento baseado em pixel é combinado com o processamento baseado em features visuais objetivando encontrar um compromisso adequado entre os desempenhos de visualização e de busca. Os resultados experimentais mostram a flexibilidade da solução proposta em alcançar diferentes compromissos de otimização, nomeadamente desempenho competitivo em relação ao padrão HEVC tanto em termos de visualização quanto de busca

Embedded landmark acquisition system for visual slam using star identification based stereo correspondence descriptor

Orientador : Prof. Dr. Eduardo TodtDissertação (mestrado) - Universidade Federal do Paraná, Setor de Ciências Exatas, Programa de Pós-Graduação em Informática. Defesa: Curitiba, 13/04/2015Inclui referênciasResumo: O uso de câmeras como sensores principais em Localização e Mapeamento Simultâneos (Simultaneous Localization and Mapping), o que é denominado SLAM Visual (Visual SLAM), tem crescido recentemente devido à queda nos preços das câmeras. Ao mesmo tempo em que imagens trazem informações mais ricas do que outros sensores típicos empregados em aplicações SLAM, como lasers e sonares, há um custo adicional de processamento significativo quando elas são utilizadas. A informação de profundidade adicional proveniente de configurações estéreo de câmeras às fazem mais interessantes para aplicações SLAM. Nesta abordagem em especial, grande parte do custo de processamento adicional vem da extração de pontos únicos ou pedaços em ambas as imagens em estéreo e da solução do problema de correspondência entre eles. Com posse dessa informação, a disparidade horizontal entre o par de imagens pode ser utilizada para recuperar a informação de profundidade. Esse trabalho explora a utilização de uma plataforma embarcada do tipo system-ona- chip (SoC) que integra um processador ARM multinúcleo com lógica FPGA como um módulo de processamento para visão estéreo. O detector de cantos Harris e Stephens (Harris & Stephens, 1988) é usado para encontrar pontos de interesse (Points of Interest, POIs) em imagens estéreo em um coprocessador soft sintetizado no FPGA para acelerar a extração de características e livrar o processador principal deste processo altamente paralelizável. As tarefas restantes tais como correção das imagens pela calibração de câmeras, encontrar um descritor único para as características detectadas e a correspondência entre os POIs no par de imagens estéreo são solucionadas em software executando no processador principal. A arquitetura proposta para o coprocessador permite que a tarefa de extração de cantos seja executada em aproximadamente metade do tempo necessário pelo processador principal sem auxílio algum. Após encontrar os POIs, para cada um dos pontos um descritor único é necessário para que seja possível encontrar o POI correspondente na outra imagem. Esse trabalho também propõe um descritor inovador que considera o relacionamento espacial bidimensional global entre os pontos detectados para descrevê-los individualmente. Para cada imagem, cada ponto da nuvem de pontos detectada pelo algoritmo de Harris e Stephens é descrito considerando-se apenas as posições relativas entre ele e seus vizinhos. Quando somente a posição é considerada, um padrão de céu estrelado noturno é formado pelos POIs. Com o padrão de POIs sendo considerado como estrelas, descritores já utilizados em problemas de identificação de estrelas podem ser reaplicados para identificar unicamente POIs. Um protótipo do descritor baseado do algoritmo de grade de Padgett e KreutzDelgado (Padgett & KreutzDelgado, 1997) é escrito e seus resultados comparados com os descritores normalmente utilizados para este propósito, mostrando que a informação espacial bidimensional pode ser utilizada por si só para resolver o problema de correspondência. O número de correspondências úteis é comparável ao atingido com o SIFT, o descritor com melhor desempenho neste quesito, enquanto a velocidade foi superior ao BRIEF, o descritor mais rápido utilizado na comparação, na plataforma ARM, com um speedup de 1,64 e 1,40 nas bases de dados dos testes. Palavras-chave: Harris; FPGA; SLAM; Hardware Reconfigurável; VHDL; Processamento de Imagem; Visão Estéreo; Computer Vision; Arquitetura Híbrida; Sistemas Embarcados; Pontos de Interesse; Keypoints; Correspondência; Correspondência Estéreo; Identificação de Estrelas; Descrição de Características; Percepção de Profundidade.Abstract: The use of cameras as the main sensors in Simultaneous Localization and Mapping, what is called Visual SLAM, has risen recently due to the fall in camera prices. While images bring richer information than other typical SLAM sensors, such as lasers and sonars, there is significant extra processing cost when they are used. The extra depth information available from stereo camera setups makes them preferable for SLAM applications. In this particular approach, great part of the added processing cost comes from extracting unique points or image patches in both stereo images and solving the correspondence problem between them. With this information, the horizontal disparity between the pair can be used to retrieve depth information. This work explores the use of an embedded system-on-a-chip (SoC) platform that integrates a multicore ARM processor with FPGA fabric as a stereo vision processing module. The Harris and Stephens corner detector (Harris & Stephens, 1988) is used to find Point of Interests (POIs) in stereo images in a hardware soft co-processor synthesized in the FPGA to speed up feature extraction and relieve this highly parallelizable process from the main embedded processor. Remaining tasks such as image correction from camera calibration, finding unique descriptor for the detected features and the correspondence between POIs in the stereo pair are solved in software running on the main processor. The proposed architecture for the co-processor enabled the corner extraction task to be performed in about half the time taken by the main processor without aid. After finding the POIs, for each point a unique descriptor is needed for finding the correspondent POI in the other image. This work also proposes an innovative descriptor that considers a global two-dimensional spatial relationship between the detected points to describe them individually. In each image, every point in the cloud of points detected by the Harris and Stephens algorithm is described by considering only the relative position between it and its neighbors. When position alone is considered, a starry night pattern is formed by the POIs. With the POI pattern being considered as stars, the descriptors already used in star identification problems can be reapplied to uniquely identify POIs. A prototype of the descriptor based on the Padgett and KreutzDelgado's grid algorithm (Padgett & KreutzDelgado, 1997) is written and the results compared with common descriptors used for this purpose, showing that two-dimensional spatial information alone can be used to solve the correspondence problem. The number of useful matches was comparable to what was obtained with SIFT, the best performing descriptor in this matter, while the speed was superior to BRIEF, the fastest descriptor used in the comparison, on the ARM platform, with a speedup of 1.64 and 1.40 on the tested datasets. Keywords: Harris; FPGA; SLAM; Reconfigurable Hardware; VHDL; Image Processing; Stereo Vision; Computer Vision; Hybrid Architecture; Embedded Systems; Point Of Interest; Keypoints; Matching; Stereo Correspondence; Star Identification; Feature Description; Depth Perception

Smart environment monitoring through micro unmanned aerial vehicles

In recent years, the improvements of small-scale Unmanned Aerial Vehicles (UAVs) in terms of flight time, automatic control, and remote transmission are promoting the development of a wide range of practical applications. In aerial video surveillance, the monitoring of broad areas still has many challenges due to the achievement of different tasks in real-time, including mosaicking, change detection, and object detection. In this thesis work, a small-scale UAV based vision system to maintain regular surveillance over target areas is proposed. The system works in two modes. The first mode allows to monitor an area of interest by performing several flights. During the first flight, it creates an incremental geo-referenced mosaic of an area of interest and classifies all the known elements (e.g., persons) found on the ground by an improved Faster R-CNN architecture previously trained. In subsequent reconnaissance flights, the system searches for any changes (e.g., disappearance of persons) that may occur in the mosaic by a histogram equalization and RGB-Local Binary Pattern (RGB-LBP) based algorithm. If present, the mosaic is updated. The second mode, allows to perform a real-time classification by using, again, our improved Faster R-CNN model, useful for time-critical operations. Thanks to different design features, the system works in real-time and performs mosaicking and change detection tasks at low-altitude, thus allowing the classification even of small objects. The proposed system was tested by using the whole set of challenging video sequences contained in the UAV Mosaicking and Change Detection (UMCD) dataset and other public datasets. The evaluation of the system by well-known performance metrics has shown remarkable results in terms of mosaic creation and updating, as well as in terms of change detection and object detection