Design of an FPGA-based smart camera and its application towards object tracking : a thesis presented in partial fulfilment of the requirements for the degree of Master of Engineering in Electronics and Computer Engineering at Massey University, Manawatu, New Zealand

Smart cameras and hardware image processing are not new concepts, yet despite the fact both have existed several decades, not much literature has been presented on the design and development process of hardware based smart cameras. This thesis will examine and demonstrate the principles needed to develop a smart camera on hardware, based on the experiences from developing an FPGA-based smart camera. The smart camera is applied on a Terasic DE0 FPGA development board, using Terasic’s 5 megapixel GPIO camera. The algorithm operates at 120 frames per second at a resolution of 640x480 by utilising a modular streaming approach. Two case studies will be explored in order to demonstrate the development techniques established in this thesis. The first case study will develop the global vision system for a robot soccer implementation. The algorithm will identify and calculate the positions and orientations of each robot and the ball. Like many robot soccer implementations each robot has colour patches on top to identify each robot and aid finding its orientation. The ball is comprised of a single solid colour that is completely distinct from the colour patches. Due to the presence of uneven light levels a YUV-like colour space labelled YC1C2 is used in order to make the colour values more light invariant. The colours are then classified using a connected components algorithm to segment the colour patches. The shapes of the classified patches are then used to identify the individual robots, and a CORDIC function is used to calculate the orientation. The second case study will investigate an improved colour segmentation design. A new HSY colour space is developed by remapping the Cartesian coordinate system from the YC1C2 to a polar coordinate system. This provides improved colour segmentation results by allowing for variations in colour value caused by uneven light patterns and changing light levels

Automation and Robotics: Latest Achievements, Challenges and Prospects

This SI presents the latest achievements, challenges and prospects for drives, actuators, sensors, controls and robot navigation with reverse validation and applications in the field of industrial automation and robotics. Automation, supported by robotics, can effectively speed up and improve production. The industrialization of complex mechatronic components, especially robots, requires a large number of special processes already in the pre-production stage provided by modelling and simulation. This area of research from the very beginning includes drives, process technology, actuators, sensors, control systems and all connections in mechatronic systems. Automation and robotics form broad-spectrum areas of research, which are tightly interconnected. To reduce costs in the pre-production stage and to reduce production preparation time, it is necessary to solve complex tasks in the form of simulation with the use of standard software products and new technologies that allow, for example, machine vision and other imaging tools to examine new physical contexts, dependencies and connections

Convergence of Intelligent Data Acquisition and Advanced Computing Systems

This book is a collection of published articles from the Sensors Special Issue on "Convergence of Intelligent Data Acquisition and Advanced Computing Systems". It includes extended versions of the conference contributions from the 10th IEEE International Conference on Intelligent Data Acquisition and Advanced Computing Systems: Technology and Applications (IDAACS’2019), Metz, France, as well as external contributions

Design of Interactive Service Robots applying methods of Systems Engineering and Decision Making

Interaktive Service Roboter werden heute bereits in einigen Anwendungsszenarien eingesetzt, in denen sie beispielsweise Menschen durch Gebäude geleiten oder bei häuslichen Aufgaben unterstützen. Dennoch gibt es bislang kein System, das den erwarteten Marktdurchbruch geschafft hat. Die hohe Komplexität solcher Systeme und vielfältige Anforderungen durch Benutzer und Betreiber erschweren die Entwicklung von erfolgreichen Service Robotern. In dieser Arbeit wurden zwei interaktive Service Roboter entwickelt, die das Potential haben, die beschriebenen Hinderungsgründe für einen breiten Einsatz zu überwinden. Das erste Robotersystem wurde als Shopping Roboter für Baumärkte entwickelt, in denen es Kunden zu gesuchten Produkten führt. Das zweite System dient als interaktiver Pflegeroboter älteren Menschen in häuslicher Umgebung bei der Bewältigung täglicher Aufgaben. Diese Arbeit beschreibt die Realisierung der Embedded Systems beider Robotersysteme und umfasst insbesondere die Entwicklung der Low-Level System Architekturen, Energie Management Systeme, Kommunikationssysteme, Sensorsysteme, sowie ausgewählte Aspekte der mechanischen Umsetzung. Die Entwicklung einer Vielzahl von Steuerungsmodulen, notwendig für die Realisierung interaktiver Service Roboter, wird beschrieben. Die vorliegende Arbeit verwendet und erweitert Methoden des Systems Engineerings, um die hohe Systemkomplexität von interaktiven Service Robotern sowie die vielfältigen Anforderungen an deren späteren Einsatz beherrschen zu können. Der Entwicklungsprozess der beiden Roboter basiert auf dem V-Model, welches einen strukturierten Entwurfsablauf unter Berücksichtigung aller Systemanforderungen erlaubt. Es zwingt ferner zur frühzeitigen Spezifikation von Prüfabläufen, was die Qualität und Zuverlässigkeit der Entwicklungsergebnisse verbessert. Für die Unterstützung von Entscheidungen im Entwicklungsprozess schlägt diese Arbeit eine Kombination aus dem V-Model und dem Analytic Hierarchy Process (AHP) vor. Der AHP hilft bei der Auswahl verfügbarer technischer Alternativen unter Berücksichtigung von Prioritäten im Entwicklungsprozess. Diese Arbeit spezifiziert sieben Kriterien, die Service Roboter charakterisieren: Anpassbarkeit, Laufzeit, Benutzbarkeit, Robustheit, Sicherheit, Features und Kosten. Die Prioritäten dieser Kriterien im Entwicklungsprozess werden für jeden Roboter individuell bestimmt. Der AHP ermittelt die beste Lösung basierend auf diesen gewichteten Kriterien und den bewerteten technischen Alternativen. Die Einbindung des AHP in den V-Model Prozess wurde am Entwurf des Shopping Roboter entwickelt und geprüft. Die Allgemeingültigkeit dieser Methode wurde während der Entwicklung des Pflegeroboters verifiziert.Interactive service robots have already been developed and operate as example installations taking over guidance tasks or serving as home assistants. However, none of these systems have become an off-the-shelf product or have achieved the predicted breakthrough so far. The challenges of the design of such systems are, on the one hand, the combination of cutting edge technologies to a complex product; on the other hand, the consideration of requirements important for the later marketing during the design process. In the framework of this dissertation, two interactive service robot systems are developed that have the potential to overcome current market entry barriers. These robots are designed to operate in two different environments: one robot guides walked-in users in large home improvement stores to requested product locations and interacts with the customer to provide product information; the other robot assists elderly people to stay longer in their homes and takes over home-care tasks. This work describes the realization of the embedded systems of both robots. In particular, the design of low-level system architectures, energy management systems, communication systems, sensor systems, and selected aspects of mechanical implementations are carried out in this work. Multiple embedded system modules are developed for the control of the robots' functionalities; the development processes as well as the composition and evaluation of these modules are presented in this work. To cope with the complexity and the various factors that are important for the design of the robots, this thesis applies and further develops system engineering methods. The development process is based on the V-Model system design method. The V-Model helps to structure the design process under consideration of all system requirements. It involves evaluation procedures at all design levels, and thus increases the quality and reliability of the development outputs. To support design decisions, this thesis proposes to combine the V-Model with the Analytic Hierarchy Process (AHP) method. The AHP helps to evaluate technical alternatives for design decisions according to overall criteria, a system has to fulfill. This thesis defines seven criteria that characterize a service robot: Adaptability, Operation Time, Usability, Robustness, Safeness, Features, and Costs. These criteria are weighted for each individual robot application. The AHP evaluates technical design alternatives based on the weighted criteria to reveal the best technical solution. The integration of the AHP into the V-Model development is tested and improved during the design process of the shopping robot system. The generality of this combined systematic design approach is validated during the design of the home-care robot system

Automatic testing of organic strain gauge tactile sensors.

Human-Robot Interaction is a developing field of science, that is posed to augment everything we do in life. Skin sensors that can detect touch, temperature, distance, and other physical interaction parameters at the human-robot interface are very important to enhancing the collaboration between humans and machines. As such, these sensors must be efficiently tested and characterized to give accurate feedback from the sensor to the robot. The objective of this work is to create a diversified software testing suite that removes as much human intervention as possible. The tests and methodology discussed here provide multiple realistic scenarios that the sensors undergo during repeated experiments. This capability allows for easy repeatable tests without interference from the test engineer, increasing productivity and efficiency. The foundation of this work has two main pieces: force feedback control to drive the test actuator, and computer vision functionality to guide alignment of the test actuator and sensors arranged in a 2D array. The software running automated tests was also made compatible with the testbench hardware via LabVIEW programs. The program uses set coordinates to complete a raster scan of the SkinCell that locates individual sensors. Tests are then applied at each sensor using a force controller. The force feedback control system uses a Proportional Integral Derivative (PID) controller that reads in force readings from a load cell to correct itself or follow a desired trajectory. The motion of the force actuator was compared to that of the projected trajectory to test for accuracy and time delay. The proposed motor control allows for dynamic force to stimulate the sensors giving a more realistic test then a stable force. A top facing camera was introduced to take in the starting position of a SkinCell before testing. Then, computer vision algorithms were proposed to extract the location of the cell and individual sensors before generating a coordinate plane. This allows for the engineer to skip over manual alignment of the sensors, saving more time and providing more accurate destinations. Finally, the testbench was applied to numerous sensors developed by the research team at the Louisville Automation and Robotics Research Institute (LARRI) for testing and data analysis. Force loads are applied to the individual sensors while recording response. Afterwards, postprocessing of the data was conducted to compare responses within the SkinCell as well as to other sensors manufactured using different methods

An embedded real-time red peach detection system based on an OV7670 camera, ARM Cortex-M4 processor and 3D Look-Up Tables

This work proposes the development of an embedded real-time fruit detection system for future automatic fruit harvesting. The proposed embedded system is based on an ARM Cortex-M4 (STM32F407VGT6) processor and an Omnivision OV7670 color camera. The future goal of this embedded vision system will be to control a robotized arm to automatically select and pick some fruit directly from the tree. The complete embedded system has been designed to be placed directly in the gripper tool of the future robotized harvesting arm. The embedded system will be able to perform real-time fruit detection and tracking by using a three-dimensional look-up-table (LUT) defined in the RGB color space and optimized for fruit picking. Additionally, two different methodologies for creating optimized 3D LUTs based on existing linear color models and fruit histograms were implemented in this work and compared for the case of red peaches. The resulting system is able to acquire general and zoomed orchard images and to update the relative tracking information of a red peach in the tree ten times per second

Kodizajn arhitekture i algoritama za lokalizacijumobilnih robota i detekciju prepreka baziranih namodelu

This thesis proposes SoPC (System on a Programmable Chip) architectures for efficient embedding of vison-based localization and obstacle detection tasks in a navigational pipeline on autonomous mobile robots. The obtained results are equivalent or better in comparison to state-ofthe- art. For localization, an efficient hardware architecture that supports EKF-SLAM's local map management with seven-dimensional landmarks in real time is developed. For obstacle detection a novel method of object recognition is proposed - detection by identification framework based on single detection window scale. This framework allows adequate algorithmic precision and execution speeds on embedded hardware platforms.Ova teza bavi se dizajnom SoPC (engl. System on a Programmable Chip) arhitektura i algoritama za efikasnu implementaciju zadataka lokalizacije i detekcije prepreka baziranih na viziji u kontekstu autonomne robotske navigacije. Za lokalizaciju, razvijena je efikasna računarska arhitektura za EKF-SLAM algoritam, koja podržava skladištenje i obradu sedmodimenzionalnih orijentira lokalne mape u realnom vremenu. Za detekciju prepreka je predložena nova metoda prepoznavanja objekata u slici putem prozora detekcije fiksne dimenzije, koja omogućava veću brzinu izvršavanja algoritma detekcije na namenskim računarskim platformama

Design and Development of Sensor Integrated Robotic Hand

Most of the automated systems using robots as agents do use few sensors according to the need. However, there are situations where the tasks carried out by the end-effector, or for that matter by the robot hand needs multiple sensors. The hand, to make the best use of these sensors, and behave autonomously, requires a set of appropriate types of sensors which could be integrated in proper manners. The present research work aims at developing a sensor integrated robot hand that can collect information related to the assigned tasks, assimilate there correctly and then do task action as appropriate. The process of development involves selection of sensors of right types and of right specification, locating then at proper places in the hand, checking their functionality individually and calibrating them for the envisaged process. Since the sensors need to be integrated so that they perform in the desired manner collectively, an integration platform is created using NI PXIe-1082. A set of algorithm is developed for achieving the integrated model. The entire process is first modelled and simulated off line for possible modification in order to ensure that all the sensors do contribute towards the autonomy of the hand for desired activity. This work also involves design of a two-fingered gripper. The design is made in such a way that it is capable of carrying out the desired tasks and can accommodate all the sensors within its fold. The developed sensor integrated hand has been put to work and its performance test has been carried out. This hand can be very useful for part assembly work in industries for any shape of part with a limit on the size of the part in mind. The broad aim is to design, model simulate and develop an advanced robotic hand. Sensors for pick up contacts pressure, force, torque, position, surface profile shape using suitable sensing elements in a robot hand are to be introduced. The hand is a complex structure with large number of degrees of freedom and has multiple sensing capabilities apart from the associated sensing assistance from other organs. The present work is envisaged to add multiple sensors to a two-fingered robotic hand having motion capabilities and constraints similar to the human hand. There has been a good amount of research and development in this field during the last two decades a lot remains to be explored and achieved. The objective of the proposed work is to design, simulate and develop a sensor integrated robotic hand. Its potential applications can be proposed for industrial environments and in healthcare field. The industrial applications include electronic assembly tasks, lighter inspection tasks, etc. Application in healthcare could be in the areas of rehabilitation and assistive techniques. The work also aims to establish the requirement of the robotic hand for the target application areas, to identify the suitable kinds and model of sensors that can be integrated on hand control system. Functioning of motors in the robotic hand and integration of appropriate sensors for the desired motion is explained for the control of the various elements of the hand. Additional sensors, capable of collecting external information and information about the object for manipulation is explored. Processes are designed using various software and hardware tools such as mathematical computation MATLAB, OpenCV library and LabVIEW 2013 DAQ system as applicable, validated theoretically and finally implemented to develop an intelligent robotic hand. The multiple smart sensors are installed on a standard six degree-of-freedom industrial robot KAWASAKI RS06L articulated manipulator, with the two-finger pneumatic SHUNK robotic hand or designed prototype and robot control programs are integrated in such a manner that allows easy application of grasping in an industrial pick-and-place operation where the characteristics of the object can vary or are unknown. The effectiveness of the actual recommended structure is usually proven simply by experiments using calibration involving sensors and manipulator. The dissertation concludes with a summary of the contribution and the scope of further work

Sensors and Technologies in Spain: State-of-the-Art

The aim of this special issue was to provide a comprehensive view on the state-of-the-art sensor technology in Spain. Different problems cause the appearance and development of new sensor technologies and vice versa, the emergence of new sensors facilitates the solution of existing real problems. [...