On the Hardware/Software Design and Implementation of a High Definition Multiview Video Surveillance System

published_or_final_versio

Hardware dedicado para sistemas empotrados de visión

La constante evolución de las Tecnologías de la Información y las Comunicaciones no solo ha permitido que más de la mitad de la población mundial esté actualmente interconectada a través de Internet, sino que ha sido el caldo de cultivo en el que han surgido nuevos paradigmas, como el ‘Internet de las cosas’ (IoT) o la ‘Inteligencia ambiental’ (AmI), que plantean la necesidad de interconectar objetos con distintas funcionalidades para lograr un entorno digital, sensible y adaptativo, que proporcione servicios de muy distinta índole a sus usuarios. La consecución de este entorno requiere el desarrollo de dispositivos electrónicos de bajo coste que, con tamaño y peso reducido, sean capaces de interactuar con el medio que los rodea, operar con máxima autonomía y proporcionar un elevado nivel de inteligencia. La funcionalidad de muchos de estos dispositivos incluirá la capacidad para adquirir, procesar y transmitir imágenes, extrayendo, interpretando o modificando la información visual que resulte de interés para una determinada aplicación. En el marco de este desafío surge la presente Tesis Doctoral, cuyo eje central es el desarrollo de hardware dedicado para la implementación de algoritmos de procesamiento de imágenes y secuencias de vídeo usados en sistemas empotrados de visión. El trabajo persigue una doble finalidad. Por una parte, la búsqueda de soluciones que, por sus prestaciones y rendimiento, puedan ser incorporadas en sistemas que satisfagan las estrictas exigencias de funcionalidad, tamaño, consumo de energía y velocidad de operación demandadas por las nuevas aplicaciones. Por otra, el diseño de una serie de bloques funcionales implementados como módulos de propiedad intelectual, que permitan aliviar la carga computacional de las unidades de procesado de los sistemas en los que se integren. En la Tesis se proponen soluciones específicas para la implementación de dos tipos de operaciones habitualmente presentes en muchos sistemas de visión artificial: la sustracción de fondo y el etiquetado de componentes conexos. Las distintas alternativas surgen como consecuencia de aplicar una adecuada relación de compromiso entre funcionalidad y coste, entendiendo este último criterio en términos de recursos de cómputo, velocidad de operación y potencia consumida, lo que permite cubrir un amplio espectro de aplicaciones. En algunas de las soluciones propuestas se han utilizado además, técnicas de inferencia basadas en Lógica Difusa con idea de mejorar la calidad de los sistemas de visión resultantes. Para la realización de los diferentes bloques funcionales se ha seguido una metodología de diseño basada en modelos, que ha permitido la realización de todo el ciclo de desarrollo en un único entorno de trabajo. Dicho entorno combina herramientas informáticas que facilitan las etapas de codificación algorítmica, diseño de circuitos, implementación física y verificación funcional y temporal de las distintas alternativas, acelerando con ello todas las fases del flujo de diseño y posibilitando una exploración más eficiente del espacio de posibles soluciones. Asimismo, con el objetivo de demostrar la funcionalidad de las distintas aportaciones de esta Tesis Doctoral, algunas de las soluciones propuestas han sido integradas en sistemas de vídeo reales, que emplean buses estándares de uso común. Los dispositivos seleccionados para llevar a cabo estos demostradores han sido FPGAs y SoPCs de Xilinx, ya que sus excelentes propiedades para el prototipado y la construcción de sistemas que combinan componentes software y hardware, los convierten en candidatos ideales para dar soporte a la implementación de este tipo de sistemas.The continuous evolution of the Information and Communication Technologies (ICT), not only has allowed more than half of the global population to be currently interconnected through Internet, but it has also been the breeding ground for new paradigms such as Internet of Things (ioT) or Ambient Intelligence (AmI). These paradigms expose the need of interconnecting elements with different functionalities in order to achieve a digital, sensitive, adaptive and responsive environment that provides services of distinct nature to the users. The development of low cost devices, with small size, light weight and a high level of autonomy, processing power and ability for interaction is required to obtain this environment. Attending to this last feature, many of these devices will include the capacity to acquire, process and transmit images, extracting, interpreting and modifying the visual information that could be of interest for a certain application. This PhD Thesis, focused on the development of dedicated hardware for the implementation of image and video processing algorithms used in embedded systems, attempts to response to this challenge. The work has a two-fold purpose: on one hand, the search of solutions that, for its performance and properties, could be integrated on systems with strict requirements of functionality, size, power consumption and speed of operation; on the other hand, the design of a set of blocks that, packaged and implemented as IP-modules, allow to alleviate the computational load of the processing units of the systems where they could be integrated. In this Thesis, specific solutions for the implementation of two kinds of usual operations in many computer vision systems are provided. These operations are background subtraction and connected component labelling. Different solutions are created as the result of applying a good performance/cost trade-off (approaching this last criteria in terms of area, speed and consumed power), able to cover a wide range of applications. Inference techniques based on Fuzzy Logic have been applied to some of the proposed solutions in order to improve the quality of the resulting systems. To obtain the mentioned solutions, a model based-design methodology has been applied. This fact has allowed us to carry out all the design flow from a single work environment. That environment combines CAD tools that facilitate the stages of code programming, circuit design, physical implementation and functional and temporal verification of the different algorithms, thus accelerating the overall processes and making it possible to explore the space of solutions. Moreover, aiming to demonstrate the functionality of this PhD Thesis’s contributions, some of the proposed solutions have been integrated on real video systems that employ common and standard buses. The devices selected to perform these demonstrators have been FPGA and SoPCs (manufactured by Xilinx) since, due to their excellent properties for prototyping and creating systems that combine software and hardware components, they are ideal to develop these applications

FPGA Implementation of Blob Recognition

Real-time embedded vision systems can be used in a wide range of applications and therefore the demand has been increasing for them. In this thesis, an FPGA-based embedded vision system capable of recognizing objects in real time is presented. The proposed system architecture consists of multiple Intellectual Properties (IPs), which are used as a set of complex instructions by an integrated 32-bit CPU Microblaze. Each IP is tailored specifically to meet the needs of the application and at the same time to consume the minimum FPGA logic resources. Integrating both hardware and software on a single FPGA chip, this system can achieve the real-time performance of full VGA video processing at 32 frames per second (fps). In addition, this work comes up with a new method called Dual Connected Component Labelling (DCCL) suitable for FPGA implementation

Identification and three-dimensional positioning of urban energy lines from optical images to aid a teleoperated pruning robot

Orientador : Prof. Dr. Leandro dos Santos CoelhoDissertação (mestrado) - Universidade Federal do Paraná, Setor de Tecnologia, Programa de Pós-Graduação em Engenharia Elétrica. Defesa: Curitiba, 30/08/2016Inclui referências : f. 138-144Área de concentraçãoResumo: Diversos fatores podem impactar a qualidade da distribuição de energia elétrica, entre eles, um dos mais impactantes é o contato de vegetação com linhas aéreas energizadas. Assim sendo, é de suma importância a poda de vegetação próxima à linhas energizadas. Visando-se aprimorar esse processo, pode-se empregar um robô teleoperado de poda, de forma que a poda possa ser realizada de maneira remota e segura. As câmeras instaladas no braço robótico permitem que o operador tenha visão da área de corte mesmo quando a visada direta do solo estiver obstruída. Um dos problemas de se visualizar a região de corte por meio de um monitor é a perda de noção de profundidade, o que pode dificultar a operação. Dessa forma, seria relevante uma técnica de visão computacional capaz de detectar as linhas de energia e seu posicionamento tridimensional (3D) a fim de auxiliar o operador. Revisando-se a literatura, avaliou-se que, no geral, os trabalhos já propostos para detecção de linhas em imagens operam em situações com fundo limpo, não urbanizado e com vista superior das linhas de energia. Assim sendo, nesse trabalho é proposta uma técnica para detecção de linhas energizadas em imagens de regiões urbanas e a obtenção de seu posicionamento 3D, fator ainda não explorado na literatura recente. Para se alcançar esse objetivo é proposta a utilização de câmeras de espectro visível posicionadas em paralelo. Assim, regiões com potencial para serem linhas de energia são selecionadas utilizando-se detecção de bordas seguidas por filtragens geométricas aplicando-se técnicas inspiradas em algoritmos de grafos e ajuste de pontos selecionados a uma curva. Após a seleção de regiões candidatas a linha de energia, o posicionamento 3D é obtido utilizando-se de visão estéreo. Para tal, a correspondência entre pontos visíveis em ambas as câmeras é encontrada e com triangulação o posicionamento 3D da linha de energia é recuperado. Com a informação 3D disponível falsos candidatos são reduzidos por um fator de aproximadamente sete vezes e finalmente as linhas são detectadas. Para avaliação do método foi criada uma base de dados contendo imagens estéreo obtidas de um cenário montado com dois postes, três linhas de energia e uma árvore entre essas, na qual foi possível atingir níveis de precisão de 98% ao término do processo de detecção, contando-se com 91% de taxa de verdadeiro positivos. As causas dos falsos negativos são evidenciadas para que trabalhos futuros possam encontrar alternativas às dificuldades apresentadas. O algoritmo aqui proposto fornece como saída um mapa de cor sobre as linhas de energia para identificação da profundidade em 2D e uma nuvem de pontos para visualização em 3D. Palavras-chave: Visão Computacional. Reconhecimento de objetos. Visão estéreo. Linhas de energia. Robô de poda.Abstract: Different factors may affect energy distribution quality, among them, one of the main causes is when vegetation gets into contact with overhead energy lines. Therefore, it is of main importance to prune vegetation close to energy lines. To improve this process it is possible to use a teleoperated robot, what allows the pruning activity to be accomplished in a remote and safe way. Cameras installed in the robot arm provide images from the pruning region to the operator even when direct sight is not an option. One of the main problems viewing the prunning region using a display is the lost of depth perception, what could make the operator unintentialy colide the robot with energy lines. Therefore, it would be of great aid a computer vision method capable of detecting energy lines and their three-dimensional (3D) positioning to aid the operator. During the state of the art review of energy line detection in images, it was perceived that, in general, the already proposed works operate in regions where the images present a clear background, not urbanized, and with the energy lines seen from above. Therefore, in this work, it is proposed a technique to detect energy lines and their 3D positioning in images taken in urban settings, factor yet unexplored in the recent literature. To reach this objective it is proposed the use of two visible spectrum cameras installed in parallel. In this way, regions with potential to be energy line are selected using edge detection followed by the geometric filtering designed using techniques inspired in graphs algorithms and curve fitting. After the regions with potential to be energy lines are found, their 3D position is obtained with stereo vision. To do so, the matching among points visible by both cameras is found and with triangulation, it is possible to recover the energy line 3D position. With the 3D information available, false positives are reduced by a factor of about seven and finally the energy lines are detected. A dataset containing stereo images of a scenario built with two power poles, three energy lines, and a tree between them was created in order to evaluate the presented method. In the commented dataset it was possible to reach accuracy of 98% at the end of the detection process, with 91% true positive rate. The causes of the false negatives cases are put in evidence in order to allow them to be overcame by future works. The algorithm proposed here outputs a colormap projected over the energy lines to inform the depth of each one in 2D and a point cloud to visualize each line in 3D. Key words: Computer vision. Object Recognition. Stereo vision. Overhead Energy Lines. Pruning Robo

Image Processing Using FPGAs

This book presents a selection of papers representing current research on using field programmable gate arrays (FPGAs) for realising image processing algorithms. These papers are reprints of papers selected for a Special Issue of the Journal of Imaging on image processing using FPGAs. A diverse range of topics is covered, including parallel soft processors, memory management, image filters, segmentation, clustering, image analysis, and image compression. Applications include traffic sign recognition for autonomous driving, cell detection for histopathology, and video compression. Collectively, they represent the current state-of-the-art on image processing using FPGAs

Ein modulares optisches Trackingsystem für medizintechnische Anwendungen: integrierte Datenflussarchitektur in Hard- und Software und Applikationsframework

Die vorliegende Arbeit beschreibt die Entwicklung eines modularen optischen Trackingsystems, ausgerichtet auf die speziellen Anforderungen im medizintechnischen Umfeld. Das Spektrum der vorgestellten Anwendungen des Systems reicht dabei von der Erfassung der Benutzerinteraktion in verschiedenen medizinischen Simulatoren (z.B. für Ophthalmochirurgie, Ophthalmoskopie und Neurochirurgie) bis hin zur Positionserfassung eines handgehaltenen Operationsroboters. Im Unterschied zu verfügbaren kommerziellen Trackingsystemem mit ihren eng umrissenen Anwendungsbereichen wird ein universell ausgelegtes Baukastensystem vorgestellt, das sich mit geringem Entwicklungsaufwand an die speziellen Anforderungen der jeweiligen Anwendungen anpassen lässt (so u.a. sehr kleine Geometrien, deformierbare Objekte, Einsatz von Originalinstrumenten, geringe Ressourcenverfügbarkeit im Simulator-PC). Zu diesem Zweck wird ein modulares Systemkonzept entwickelt, welches von der spezialisierten Datenverarbeitung gängiger Trackingsysteme abstrahiert und auf einer generalisierten, modularen Systemarchitektur für den Einsatz aller Arten von Markern mit drei Freiheitsgraden aufbaut. Neben den verbreiteten infrarotbasierten Signaliserungstechniken werden dabei auch passive Farbmarker zur Objektsignalisierung unterstützt. Die Implementierung von Bildverarbeitungsaufgaben in spezialisierter Hardware (FPGAs) direkt auf dem Kameradatenstrom ermöglicht eine frühzeitige Datenreduktion und damit niedrige Latenzzeiten. Der Entwicklungsprozess für neuartige Trackinglösungen wird vereinfacht durch die enge Integration der Hard- und Softwaremodule in einer einheitlichen durchgängigen Datenflussarchitektur, die flexibel an die jeweilige Aufgabenstellung anpassbar ist. Ein erweiterbares graphisches Frontend schließlich unterstützt bei Betrieb und Konfiguration und erlaubt auch die Simulation ganzer Systeme während der Entwicklung

International Workshop on MicroFactories (IWMF 2012): 17th-20th June 2012 Tampere Hall Tampere, Finland

This Workshop provides a forum for researchers and practitioners in industry working on the diverse issues of micro and desktop factories, as well as technologies and processes applicable for micro and desktop factories. Micro and desktop factories decrease the need of factory floor space, and reduce energy consumption and improve material and resource utilization thus strongly supporting the new sustainable manufacturing paradigm. They can be seen also as a proper solution to point-of-need manufacturing of customized and personalized products near the point of need