FPGA-based module for SURF extraction

We present a complete hardware and software solution of an FPGA-based computer vision embedded module capable of carrying out SURF image features extraction algorithm. Aside from image analysis, the module embeds a Linux distribution that allows to run programs specifically tailored for particular applications. The module is based on a Virtex-5 FXT FPGA which features powerful configurable logic and an embedded PowerPC processor. We describe the module hardware as well as the custom FPGA image processing cores that implement the algorithm's most computationally expensive process, the interest point detection. The module's overall performance is evaluated and compared to CPU and GPU based solutions. Results show that the embedded module achieves comparable disctinctiveness to the SURF software implementation running in a standard CPU while being faster and consuming significantly less power and space. Thus, it allows to use the SURF algorithm in applications with power and spatial constraints, such as autonomous navigation of small mobile robots

Deep Virtual Networks for Memory Efficient Inference of Multiple Tasks

Deep networks consume a large amount of memory by their nature. A natural question arises can we reduce that memory requirement whilst maintaining performance. In particular, in this work we address the problem of memory efficient learning for multiple tasks. To this end, we propose a novel network architecture producing multiple networks of different configurations, termed deep virtual networks (DVNs), for different tasks. Each DVN is specialized for a single task and structured hierarchically. The hierarchical structure, which contains multiple levels of hierarchy corresponding to different numbers of parameters, enables multiple inference for different memory budgets. The building block of a deep virtual network is based on a disjoint collection of parameters of a network, which we call a unit. The lowest level of hierarchy in a deep virtual network is a unit, and higher levels of hierarchy contain lower levels' units and other additional units. Given a budget on the number of parameters, a different level of a deep virtual network can be chosen to perform the task. A unit can be shared by different DVNs, allowing multiple DVNs in a single network. In addition, shared units provide assistance to the target task with additional knowledge learned from another tasks. This cooperative configuration of DVNs makes it possible to handle different tasks in a memory-aware manner. Our experiments show that the proposed method outperforms existing approaches for multiple tasks. Notably, ours is more efficient than others as it allows memory-aware inference for all tasks.Comment: CVPR 201

Applications of Intelligent Vision in Low-Cost Mobile Robots

With the development of intelligent information technology, we have entered an era of 5G and AI. Mobile robots embody both of these technologies, and as such play an important role in future developments. However, the development of perception vision in consumer-grade low-cost mobile robots is still in its infancies. With the popularity of edge computing technology in the future, high-performance vision perception algorithms are expected to be deployed on low-power edge computing chips. Within the context of low-cost mobile robotic solutions, a robot intelligent vision system is studied and developed in this thesis. The thesis proposes and designs the overall framework of the higher-level intelligent vision system. The core system includes automatic robot navigation and obstacle object detection. The core algorithm deployments are implemented through a low-power embedded platform. The thesis analyzes and investigates deep learning neural network algorithms for obstacle object detection in intelligent vision systems. By comparing a variety of open source object detection neural networks on high performance hardware platforms, combining the constraints of hardware platform, a suitable neural network algorithm is selected. The thesis combines the characteristics and constraints of the low-power hardware platform to further optimize the selected neural network. It introduces the minimize mean square error (MMSE) and the moving average minmax algorithms in the quantization process to reduce the accuracy loss of the quantized model. The results show that the optimized neural network achieves a 20-fold improvement in inference performance on the RK3399PRO hardware platform compared to the original network. The thesis concludes with the application of the above modules and systems to a higher-level intelligent vision system for a low-cost disinfection robot, and further optimization is done for the hardware platform. The test results show that while achieving the basic service functions, the robot can accurately identify the obstacles ahead and locate and navigate in real time, which greatly enhances the perception function of the low-cost mobile robot

FPGA synthesis of an stereo image matching architecture for autonomous mobile robots

This paper describes a hardware proposal to speed up the process of image matching in stereo vision systems like those employed by autonomous mobile robots. This proposal combines a classical window-based matching approach with a previous stage, where key points are selected from each image of the stereo pair. In this first step the key point extraction method is based on the SIFT algorithm. Thus, in the second step, the window-based matching is only applied to the set of selected key points, instead of to the whole images. For images with a 1% of key points, this method speeds up the matching four orders of magnitude. This proposal is, on the one hand, a better parallelizable architecture than the original SIFT, and on the other, a faster technique than a full image windows matching approach. The architecture has been implemented on a lower power Virtex 6 FPGA and it achieves a image matching speed above 30 fps.This work has been funded by Spanish government project TEC2015-66878-C3-2-R (MINECO/FEDER, UE)

A Cost-Effective Person-Following System for Assistive Unmanned Vehicles with Deep Learning at the Edge

The vital statistics of the last century highlight a sharp increment of the average age of the world population with a consequent growth of the number of older people. Service robotics applications have the potentiality to provide systems and tools to support the autonomous and self-sufficient older adults in their houses in everyday life, thereby avoiding the task of monitoring them with third parties. In this context, we propose a cost-effective modular solution to detect and follow a person in an indoor, domestic environment. We exploited the latest advancements in deep learning optimization techniques, and we compared different neural network accelerators to provide a robust and flexible person-following system at the edge. Our proposed cost-effective and power-efficient solution is fully-integrable with pre-existing navigation stacks and creates the foundations for the development of fully-autonomous and self-contained service robotics applications

A survey on real-time 3D scene reconstruction with SLAM methods in embedded systems

The 3D reconstruction of simultaneous localization and mapping (SLAM) is an important topic in the field for transport systems such as drones, service robots and mobile AR/VR devices. Compared to a point cloud representation, the 3D reconstruction based on meshes and voxels is particularly useful for high-level functions, like obstacle avoidance or interaction with the physical environment. This article reviews the implementation of a visual-based 3D scene reconstruction pipeline on resource-constrained hardware platforms. Real-time performances, memory management and low power consumption are critical for embedded systems. A conventional SLAM pipeline from sensors to 3D reconstruction is described, including the potential use of deep learning. The implementation of advanced functions with limited resources is detailed. Recent systems propose the embedded implementation of 3D reconstruction methods with different granularities. The trade-off between required accuracy and resource consumption for real-time localization and reconstruction is one of the open research questions identified and discussed in this paper

Machine Learning Algorithms for Robotic Navigation and Perception and Embedded Implementation Techniques

L'abstract è presente nell'allegato / the abstract is in the attachmen

Inspecção Visual de Isoladores Eléctricos -Abordagem baseada em Deep Learning

To supply the electrical population’s demand is necessary to have a good quality power distribution systems. Electrical asset inspection, like electrical towers, dam or power line is a high risk and expensive task. Nowadays it is done with traditional methods like using a helicopter equipped with several sensors or with specialised human labour. In the last years, the Unmanned Aerial Vehicle (UAV) exponential growth (most common called drones) make them very accessible for different applications. They are cheaper and easy to adapt. Adopting this technology will be in the future the next step on electrical asset inspection. It will provide a better service (safer, faster and cheaper), particularly in power line distribution. This thesis brings forward an alternative to traditional methods using a UAV for images processing during the insulator visual inspection. The developed work implement real-time insulators visual detection using na Artificial Neural Network (ANN), You Only Look Once (YOLO) in this case, on medium and high voltage power lines. YOLO was trained with different types and sizes of insulators. Isn’t always possible to see what the UAV is recording so it has a gimbal system which controls the camera orientation/position. It will centre the insulator on the image and this way getting a better view of it. All the training and tests were performed on board Jetson TX2.A inspeção de ativos elétricos, sejam eles torres elétricas, barragens ou linhas elétricas, é realizada com recurso a helicópteros, equipados com sensores para o efeito ou, de uma forma mais minuciosa, com o recurso a mão-de-obra especializada. Ambas as situações são trabalhos de risco elevado. Nos últimos anos temos assistido a um enorme crescimento de veículos aéreos não tripulados, vulgarmente chamados de drones. Estes sistemas estão bastante desenvolvidos e são economicamente acessíveis, o que os torna perfeitos para variadíssimas funções. A inspeção de linhas elétricas não ´e exceção. Esta dissertação, pretende ser uma primeira abordagem `a utilização de drones para uma inspeção autónoma de linhas elétricas, nomeadamente no processamento de imagem para inspeção visual de isoladores. O trabalho desenvolvido, consiste na implementação de um sistema que funciona em tempo real para a deteção visual de isoladores. A deteção ´e feita com recurso a uma rede neuronal, neste caso específico a fico a You Only Look Once (YOLO), que foi treinada com isoladores de diferentes tamanhos e materiais. Uma vez que nem sempre ´e possível acompanhar o que está a ser filmado, o drone consta de um sistema capaz de orientar a câmara, chamado gimbal, para centrar o isolador na imagem e assim conseguir obter um melhor enquadramento do ativo a ser inspecionado. Todos este desenvolvimentos e consequentes testes foram realizados com a utilização de processamento paralelo, que neste caso foi utilizada a placa Jetson TX2