Implementation of a Bacterium Tracking System on FPGA

ABSTRACT Tracking in computer vision is monitoring the movement of an object in a sequence of images. This object tracking may be performed for different reasons such as security systems, animation production, etc. Hundreds of publications in the form of books, articles, and journal papers indicate the importance of tracking as the research domain in universities and research centers. Tracking consists of the answer to two principal problems, the motion problem and the matching problem. Two solutions for the motion problem are adjacent regions method and the Kalman filter. Many solutions also exist for matching problem such as window tracking, detection of the moving object by its specifications like edges, corners and contour, detection of the target by simple shapes like circles, squares, rectangles and triangles, detection by 3-dimensional shapes like cylinders and finally matching with complex shapes like human, vehicle, etc. The implemented tracking algorithm in this project applies the adjacent regions for motion detection and window tracking for matching. Three different algorithms are simulated by Matlab software and one of them is implemented on the target hardware. The subject of tracking in this project is Magnetotactic Bacterium (MTB). MTB is applied in medial and biomedical fields. As an example, they can carry necessary materials into the patient‟s vascular system, where only anoparticles are able to travel. The hardware of this project is the Xilinx ML402 video starter kit, camera and microscope. This hardware system consists of the ML402 main board and the VIODC daughter board. The VIODC contains a Virtex-II FPGA which holds the drivers of different video input and output interfaces. The ML402 has a Virtex-4 FPGA and developers can implement their designs on this FPGA chip. The camera captures the video frames and the VIODC receives the frames from camera and sends them to ML402. ML402 executes the tracking algorithm on the received frames and sends the appropriate output to VIODC to be displayed. The basic module of the tracking system which is called “vid_tracking” processor core (pcore) is available with the Simulink library of the Matlab software. It can also be simulated either by real-time video or by simulated video signals. Vid_tracking consists of EDK processor and System Generator blocks. System Generator translates the pcore to physical hardware. Through the translation process, the EDK processor of Simulink library is implemented as a MicroBlaze processor which is the standard processor architecture of Virtex-4 FPGAs. The design is then exported into the XPS platform. The system developer can modify the design using the VHDL hardware description language. VHDL is very efficient and highly flexible to implement hardware systems with different abstraction levels. We applied three DSP48 modules from the Virtex-4 library in order to implement the multiplication operation in a more efficient manner. This library contains some already designed components for different Xilinx FPGAs. These components are very fast and easy to use. In order to implement the tracking algorithm, we profit from three synchronization signals which are pixel enable, horizontal synchronization and vertical synchronization. Pixel enable indicates when a new pixel is detected by the VIODC. The horizontal synchronization signal indicates the start of a line and the vertical synchronization shows the start of a frame. The tracking algorithm counts the pixels and lines to find a window of about 20-pixel by 20-pixel in the middle of the frame. It looks for a bacterium by comparing the pixel‟s intensity. When it finds a bacterium, it looks for it in the 20-pixel by 20- pixel regions around the previous window in the next frame. The Matlab simulation of MTB tracking is 100% robust since the bacterium was not missed in any of the frames during the tracking and the algorithm is able to follow the trajectory of the bacterium during all the simulation time. The simulation indicates 100% precision in 2-dimensional movements of the bacterium, since all the pixels which are recognized by the human eye as the bacterium pixels are also detected as the bacterium pixels by the simulator algorithm. The designed hardware detects whether a bacterium is in camera field of view or not. The bacterium moves at an average speed between 180 to 240 micrometers per second. A speed of 300 μm/s has also been recorded for this type of bacterium. Our tracking system could detect the moving bacterium at either of these speeds. ----------RÉSUMÉ Dans le domaine de la vision par ordinateur, le suivi est la surveillance du mouvement d'un objet dans une séquence d'images. Ce suivi d‟objet peut être accompli dans différentes applications comme les systèmes de sûreté, la production de dessins animés, etc. Des centaines de publications sous forme de livres, articles et documents de journaux indiquent l'importance du suivi dans la recherche académique et dans les centres de recherche. Le suivi contient la réponse à deux problèmes principaux, le problème du mouvement et le problème de la correspondance. Deux solutions pour le problème du mouvement sont la méthode des régions contiguës et le filtre de Kalman. D‟autres solutions existent également pour le problème de correspondance, par exemple le suivi par fenêtre, la détection d‟objet mobile par des spécifications comme des arrêtes, des coins et le contour, la détection d‟objet par des formes simples comme des cercles, des carrés et des rectangles, la détection par des formes en 3-dimensions comme des cylindres, et finalement la correspondance par des formes complexes comme l‟humain, le véhicule, etc. L'algorithme de suivi dans ce projet applique la méthode des régions contiguës pour le problème du mouvement et la méthode de suivi par fenêtre pour le problème de correspondance. Le sujet du suivi de ce projet est la bactérie magnétotactique (MTB). Les MTB sont utilisées dans des applications médicales. Par exemple, elles peuvent amener les médicaments nécessaires dans le système vasculaire du patient, où seulement des nanoparticules sont capables de se rendre. Le matériel utilisé dans ce projet est la planchette de développement pour les applications de vidéo ML402 de la compagnie Xilinx, une caméra et un microscope. Ce système matériel contient le circuit principal ML402 et la planchette VIODC. La VIODC contient un FPGA Virtex-II qui comprend des pilotes pour différentes interfaces de l'entrée et de la sortie vidéo. La ML402 est dotée d'un FPGA Virtex-4 sur laquelle les développeurs peuvent implémenter leurs designs. La VIODC reçoit les trames vidéo capturées par la caméra et les envoie vers la ML402. Cette dernière exécute l'algorithme de suivi sur les trames reçues et envoie la sortie appropriée vers la VIODC pour être affichée. Le composant principal du système de suivi qui s'appelle le processeur coeur (pcore) “vid_tracking” est disponible dans la librairie Simulink du logiciel Matlab. Ce composant peut être simulé soit par vidéo en temps-réel soit par des signaux simulés de vidéo. Le vid_tracking contient le processeur EDK et des blocs de générateur de système. Le bloc générateur de système traduit le pcore en du matériel physique. Pendant le processus de la traduction, le processeur EDK de la librairie Simulink est implémenté comme un processeur MicroBlaze qui est l'architecture de processeur standard des FPGA Virtex-4. Ensuite, le design est exporté à la plateforme XPS. Le développeur de système peut modifier le design en utilisant le langage de description matériel VHDL. VHDL est très efficace et extrêmement flexible pour implémenter des systèmes matériels avec des niveaux d'abstraction différents. On a appliqué trois modules de DSP48 de la librairie Virtex-4 pour implémenter la multiplication d'une manière plus efficace. Cette librairie contient des composants déjà conçus pour différents FPGAs de Xilinx. Ces composants sont très rapides et faciles à utiliser. Pour implémenter l'algorithme de suivi, on profite de trois signaux de synchronisation : -pixel actif, synchronisation horizontale et synchronisation verticale. Le signal pixel actif indique la détection d‟un nouveau pixel par la VIODC. Le signal de la synchronisation horizontale indique le début d‟une ligne et le signal de la synchronisation verticale montre le début d‟une trame. L'algorithme de suivi compte le nombre des pixels et le nombre de lignes pour trouver une fenêtre de 20-pixel par 20-pixel au milieu de la trame. Il cherche une bactérie par comparaison des intensités de pixels. Quand il trouve une bactérie, il la recherche dans les régions de la taille 20-pixel par 20-pixel dans les fenêtres qui suivent au voisinage de la fenêtre précédente. La simulation Matlab de suivi de MTB est 100% robuste car la bactérie n‟a été perdue dans aucune trame pendant le suivi de bactérie. L‟algorithme est capable de suivre la trajectoire de la bactérie pendant toute la durée de la simulation. La simulation atteint 100% de précision dans le cas de mouvement à 2-dimensions de la bactérie en comparant les résultats obtenus par l'algorithme à ceux obtenus par un observateur humain. Le matériel conçu détecte si une bactérie est dans le champ de vue de la caméra ou non. La bactérie se déplace avec une vitesse moyenne de 180 à 240 micromètres par seconde. Une vitesse de 300 μm/s a été aussi enregistrée pour ce type de bactérie. Notre système de suivi peut détecter la bactérie qui se déplace à n'importe quelle vitesse dans la gamme spécifiée ci-dessus

Advanced Knowledge Application in Practice

The integration and interdependency of the world economy leads towards the creation of a global market that offers more opportunities, but is also more complex and competitive than ever before. Therefore widespread research activity is necessary if one is to remain successful on the market. This book is the result of research and development activities from a number of researchers worldwide, covering concrete fields of research

Ubiquitous supercomputing : design and development of enabling technologies for multi-robot systems rethinking supercomputing

Supercomputing, also known as High Performance Computing (HPC), is almost everywhere (ubiquitous), from the small widget in your phone telling you that today will be a sunny day, up to the next great contribution to the understanding of the origins of the universe.However, there is a field where supercomputing has been only slightly explored - robotics. Other than attempts to optimize complex robotics tasks, the two forces lack an effective alignment and a purposeful long-term contract. With advancements in miniaturization, communications and the appearance of powerful, energy and weight optimized embedded computing boards, a next logical transition corresponds to the creation of clusters of robots, a set of robotic entities that behave similarly as a supercomputer does. Yet, there is key aspect regarding our current understanding of what supercomputing means, or is useful for, that this work aims to redefine. For decades, supercomputing has been solely intended as a computing efficiency mechanism i.e. decreasing the computing time for complex tasks. While such train of thought have led to countless findings, supercomputing is more than that, because in order to provide the capacity of solving most problems quickly, another complete set of features must be provided, a set of features that can also be exploited in contexts such as robotics and that ultimately transform a set of independent entities into a cohesive unit.This thesis aims at rethinking what supercomputing means and to devise strategies to effectively set its inclusion within the robotics realm, contributing therefore to the ubiquity of supercomputing, the first main ideal of this work. With this in mind, a state of the art concerning previous attempts to mix robotics and HPC will be outlined, followed by the proposal of High Performance Robotic Computing (HPRC), a new concept mapping supercomputing to the nuances of multi-robot systems. HPRC can be thought as supercomputing in the edge and while this approach will provide all kind of advantages, in certain applications it might not be enough since interaction with external infrastructures will be required or desired. To facilitate such interaction, this thesis proposes the concept of ubiquitous supercomputing as the union of HPC, HPRC and two more type of entities, computing-less devices (e.g. sensor networks, etc.) and humans.The results of this thesis include the ubiquitous supercomputing ontology and an enabling technology depicted as The ARCHADE. The technology serves as a middleware between a mission and a supercomputing infrastructure and as a framework to facilitate the execution of any type of mission, i.e. precision agriculture, entertainment, inspection and monitoring, etc. Furthermore, the results of the execution of a set of missions are discussed.By integrating supercomputing and robotics, a second ideal is targeted, ubiquitous robotics, i.e. the use of robots in all kind of applications. Correspondingly, a review of existing ubiquitous robotics frameworks is presented and based upon its conclusions, The ARCHADE's design and development have followed the guidelines for current and future solutions. Furthermore, The ARCHADE is based on a rethought supercomputing where performance is not the only feature to be provided by ubiquitous supercomputing systems. However, performance indicators will be discussed, along with those related to other supercomputing features.Supercomputing has been an excellent ally for scientific exploration and not so long ago for commercial activities, leading to all kind of improvements in our lives, in our society and in our future. With the results of this thesis, the joining of two fields, two forces previously disconnected because of their philosophical approaches and their divergent backgrounds, holds enormous potential to open up our imagination for all kind of new applications and for a world where robotics and supercomputing are everywhere.La supercomputación, también conocida como Computación de Alto Rendimiento (HPC por sus siglas en inglés) puede encontrarse en casi cualquier lugar (ubicua), desde el widget en tu teléfono diciéndote que hoy será un día soleado, hasta la siguiente gran contribución al entendimiento de los orígenes del universo. Sin embargo, hay un campo en el que ha sido poco explorada - la robótica. Más allá de intentos de optimizar tareas robóticas complejas, las dos fuerzas carecen de un contrato a largo plazo. Dado los avances en miniaturización, comunicaciones y la aparición de potentes computadores embebidos, optimizados en peso y energía, la siguiente transición corresponde a la creación de un cluster de robots, un conjunto de robots que se comportan de manera similar a un supercomputador. No obstante, hay un aspecto clave, con respecto a la comprensión de la supercomputación, que esta tesis pretende redefinir. Durante décadas, la supercomputación ha sido entendida como un mecanismo de eficiencia computacional, es decir para reducir el tiempo de computación de ciertos problemas extremadamente complejos. Si bien este enfoque ha conducido a innumerables hallazgos, la supercomputación es más que eso, porque para proporcionar la capacidad de resolver todo tipo de problemas rápidamente, se debe proporcionar otro conjunto de características que también pueden ser explotadas en la robótica y que transforman un conjunto de robots en una unidad cohesiva. Esta tesis pretende repensar lo que significa la supercomputación y diseñar estrategias para establecer su inclusión dentro del mundo de la robótica, contribuyendo así a su ubicuidad, el principal ideal de este trabajo. Con esto en mente, se presentará un estado del arte relacionado con intentos anteriores de mezclar robótica y HPC, seguido de la propuesta de Computación Robótica de Alto Rendimiento (HPRC, por sus siglas en inglés), un nuevo concepto, que mapea la supercomputación a los matices específicos de los sistemas multi-robot. HPRC puede pensarse como supercomputación en el borde y si bien este enfoque proporcionará todo tipo de ventajas, ciertas aplicaciones requerirán una interacción con infraestructuras externas. Para facilitar dicha interacción, esta tesis propone el concepto de supercomputación ubicua como la unión de HPC, HPRC y dos tipos más de entidades, dispositivos sin computación embebida y seres humanos. Los resultados de esta tesis incluyen la ontología de la supercomputación ubicua y una tecnología llamada The ARCHADE. La tecnología actúa como middleware entre una misión y una infraestructura de supercomputación y como framework para facilitar la ejecución de cualquier tipo de misión, por ejemplo, agricultura de precisión, inspección y monitoreo, etc. Al integrar la supercomputación y la robótica, se busca un segundo ideal, robótica ubicua, es decir el uso de robots en todo tipo de aplicaciones. Correspondientemente, una revisión de frameworks existentes relacionados serán discutidos. El diseño y desarrollo de The ARCHADE ha seguido las pautas y sugerencias encontradas en dicha revisión. Además, The ARCHADE se basa en una supercomputación repensada donde la eficiencia computacional no es la única característica proporcionada a sistemas basados en la tecnología. Sin embargo, se analizarán indicadores de eficiencia computacional, junto con otros indicadores relacionados con otras características de la supercomputación. La supercomputación ha sido un excelente aliado para la exploración científica, conduciendo a todo tipo de mejoras en nuestras vidas, nuestra sociedad y nuestro futuro. Con los resultados de esta tesis, la unión de dos campos, dos fuerzas previamente desconectadas debido a sus enfoques filosóficos y sus antecedentes divergentes, tiene un enorme potencial para abrir nuestra imaginación hacia todo tipo de aplicaciones nuevas y para un mundo donde la robótica y la supercomputación estén en todos lado

Ubiquitous supercomputing : design and development of enabling technologies for multi-robot systems rethinking supercomputing

Supercomputing, also known as High Performance Computing (HPC), is almost everywhere (ubiquitous), from the small widget in your phone telling you that today will be a sunny day, up to the next great contribution to the understanding of the origins of the universe.However, there is a field where supercomputing has been only slightly explored - robotics. Other than attempts to optimize complex robotics tasks, the two forces lack an effective alignment and a purposeful long-term contract. With advancements in miniaturization, communications and the appearance of powerful, energy and weight optimized embedded computing boards, a next logical transition corresponds to the creation of clusters of robots, a set of robotic entities that behave similarly as a supercomputer does. Yet, there is key aspect regarding our current understanding of what supercomputing means, or is useful for, that this work aims to redefine. For decades, supercomputing has been solely intended as a computing efficiency mechanism i.e. decreasing the computing time for complex tasks. While such train of thought have led to countless findings, supercomputing is more than that, because in order to provide the capacity of solving most problems quickly, another complete set of features must be provided, a set of features that can also be exploited in contexts such as robotics and that ultimately transform a set of independent entities into a cohesive unit.This thesis aims at rethinking what supercomputing means and to devise strategies to effectively set its inclusion within the robotics realm, contributing therefore to the ubiquity of supercomputing, the first main ideal of this work. With this in mind, a state of the art concerning previous attempts to mix robotics and HPC will be outlined, followed by the proposal of High Performance Robotic Computing (HPRC), a new concept mapping supercomputing to the nuances of multi-robot systems. HPRC can be thought as supercomputing in the edge and while this approach will provide all kind of advantages, in certain applications it might not be enough since interaction with external infrastructures will be required or desired. To facilitate such interaction, this thesis proposes the concept of ubiquitous supercomputing as the union of HPC, HPRC and two more type of entities, computing-less devices (e.g. sensor networks, etc.) and humans.The results of this thesis include the ubiquitous supercomputing ontology and an enabling technology depicted as The ARCHADE. The technology serves as a middleware between a mission and a supercomputing infrastructure and as a framework to facilitate the execution of any type of mission, i.e. precision agriculture, entertainment, inspection and monitoring, etc. Furthermore, the results of the execution of a set of missions are discussed.By integrating supercomputing and robotics, a second ideal is targeted, ubiquitous robotics, i.e. the use of robots in all kind of applications. Correspondingly, a review of existing ubiquitous robotics frameworks is presented and based upon its conclusions, The ARCHADE's design and development have followed the guidelines for current and future solutions. Furthermore, The ARCHADE is based on a rethought supercomputing where performance is not the only feature to be provided by ubiquitous supercomputing systems. However, performance indicators will be discussed, along with those related to other supercomputing features.Supercomputing has been an excellent ally for scientific exploration and not so long ago for commercial activities, leading to all kind of improvements in our lives, in our society and in our future. With the results of this thesis, the joining of two fields, two forces previously disconnected because of their philosophical approaches and their divergent backgrounds, holds enormous potential to open up our imagination for all kind of new applications and for a world where robotics and supercomputing are everywhere.La supercomputación, también conocida como Computación de Alto Rendimiento (HPC por sus siglas en inglés) puede encontrarse en casi cualquier lugar (ubicua), desde el widget en tu teléfono diciéndote que hoy será un día soleado, hasta la siguiente gran contribución al entendimiento de los orígenes del universo. Sin embargo, hay un campo en el que ha sido poco explorada - la robótica. Más allá de intentos de optimizar tareas robóticas complejas, las dos fuerzas carecen de un contrato a largo plazo. Dado los avances en miniaturización, comunicaciones y la aparición de potentes computadores embebidos, optimizados en peso y energía, la siguiente transición corresponde a la creación de un cluster de robots, un conjunto de robots que se comportan de manera similar a un supercomputador. No obstante, hay un aspecto clave, con respecto a la comprensión de la supercomputación, que esta tesis pretende redefinir. Durante décadas, la supercomputación ha sido entendida como un mecanismo de eficiencia computacional, es decir para reducir el tiempo de computación de ciertos problemas extremadamente complejos. Si bien este enfoque ha conducido a innumerables hallazgos, la supercomputación es más que eso, porque para proporcionar la capacidad de resolver todo tipo de problemas rápidamente, se debe proporcionar otro conjunto de características que también pueden ser explotadas en la robótica y que transforman un conjunto de robots en una unidad cohesiva. Esta tesis pretende repensar lo que significa la supercomputación y diseñar estrategias para establecer su inclusión dentro del mundo de la robótica, contribuyendo así a su ubicuidad, el principal ideal de este trabajo. Con esto en mente, se presentará un estado del arte relacionado con intentos anteriores de mezclar robótica y HPC, seguido de la propuesta de Computación Robótica de Alto Rendimiento (HPRC, por sus siglas en inglés), un nuevo concepto, que mapea la supercomputación a los matices específicos de los sistemas multi-robot. HPRC puede pensarse como supercomputación en el borde y si bien este enfoque proporcionará todo tipo de ventajas, ciertas aplicaciones requerirán una interacción con infraestructuras externas. Para facilitar dicha interacción, esta tesis propone el concepto de supercomputación ubicua como la unión de HPC, HPRC y dos tipos más de entidades, dispositivos sin computación embebida y seres humanos. Los resultados de esta tesis incluyen la ontología de la supercomputación ubicua y una tecnología llamada The ARCHADE. La tecnología actúa como middleware entre una misión y una infraestructura de supercomputación y como framework para facilitar la ejecución de cualquier tipo de misión, por ejemplo, agricultura de precisión, inspección y monitoreo, etc. Al integrar la supercomputación y la robótica, se busca un segundo ideal, robótica ubicua, es decir el uso de robots en todo tipo de aplicaciones. Correspondientemente, una revisión de frameworks existentes relacionados serán discutidos. El diseño y desarrollo de The ARCHADE ha seguido las pautas y sugerencias encontradas en dicha revisión. Además, The ARCHADE se basa en una supercomputación repensada donde la eficiencia computacional no es la única característica proporcionada a sistemas basados en la tecnología. Sin embargo, se analizarán indicadores de eficiencia computacional, junto con otros indicadores relacionados con otras características de la supercomputación. La supercomputación ha sido un excelente aliado para la exploración científica, conduciendo a todo tipo de mejoras en nuestras vidas, nuestra sociedad y nuestro futuro. Con los resultados de esta tesis, la unión de dos campos, dos fuerzas previamente desconectadas debido a sus enfoques filosóficos y sus antecedentes divergentes, tiene un enorme potencial para abrir nuestra imaginación hacia todo tipo de aplicaciones nuevas y para un mundo donde la robótica y la supercomputación estén en todos ladosPostprint (published version

Mobile Robotics

The book is a collection of ten scholarly articles and reports of experiences and perceptions concerning pedagogical practices with mobile robotics.“This work is funded by CIEd – Research Centre on Education, project UID/CED/01661/2019, Institute of Education, University of Minho, through national funds of FCT/MCTES-PT.

A ROS-based software architecture for a versatile collaborative dual-armed autonomous mobile robot for the manufacturing industry

The industrial context is changing rapidly due to advancements in technology fueled by the Internet and Information Technology. The fourth industrial revolution counts integration, flexibility, and optimization as its fundamental pillars, and, in this context, Human-Robot Collaboration has become a crucial factor for manufacturing sustainability in Europe. Collaborative robots are appealing to many companies due to their low installation and running costs and high degree of flexibility, making them ideal for reshoring production facilities with a short return on investment. The ROSSINI European project aims to implement a true Human-Robot Collaboration by designing, developing, and demonstrating a modular and scalable platform for integrating human-centred robotic technologies in industrial production environments. The project focuses on safety concerns related to introducing a cobot in a shared working area and aims to lay the groundwork for a new working paradigm at the industrial level. The need for a software architecture suitable to the robotic platform employed in one of three use cases selected to deploy and test the new technology was the main trigger of this Thesis. The chosen application consists of the automatic loading and unloading of raw-material reels to an automatic packaging machine through an Autonomous Mobile Robot composed of an Autonomous Guided Vehicle, two collaborative manipulators, and an eye-on-hand vision system for performing tasks in a partially unstructured environment. The results obtained during the ROSSINI use case development were later used in the SENECA project, which addresses the need for robot-driven automatic cleaning of pharmaceutical bins in a very specific industrial context. The inherent versatility of mobile collaborative robots is evident from their deployment in the two projects with few hardware and software adjustments. The positive impact of Human-Robot Collaboration on diverse production lines is a motivation for future investments in research on this increasingly popular field by the industry

Swarm Robotics

Collectively working robot teams can solve a problem more efficiently than a single robot, while also providing robustness and flexibility to the group. Swarm robotics model is a key component of a cooperative algorithm that controls the behaviors and interactions of all individuals. The robots in the swarm should have some basic functions, such as sensing, communicating, and monitoring, and satisfy the following properties