Self-Adaptive Architecture for Multi-sensor Embedded Vision System

International audienceArchitectural optimization for heterogeneous multi-sensor processing is a real technological challenge. Most of the vision systems involve only one single color sensor and they do not address the heterogeneous sensors challenge. However, more and more applications require other types of sensor in addition, such as infrared or low-light sensor, so that the vision system could face various luminosity conditions. These heterogeneous sensors could differ in the spectral band, the resolution or even the frame rate. Such sensor variety needs huge computing performance , but embedded systems have stringent area and power constraints. Reconfigurable architecture makes possible flexible computing while respecting the latter constraints. Many reconfigurable architectures for vision application have been proposed in the past. Yet, few of them propose a real dynamic adaptation capability to manage sensor heterogeneity. In this paper, a self-adaptive architecture is proposed to deal with heterogeneous sensors dynamically. This architecture supports on-the-fly sensor switch. Architecture of the system is self-adapted thanks to a system monitor and an adaptation controller. A stream header concept is used to convey sensor information to the self-adaptive architecture. The proposed architecture was implemented in Altera Cyclone V FPGA. In this implementation, adaptation of the architecture consists in Dynamic and Partial Reconfiguration of FPGA. The self-adaptive ability of the architecture has been proved with low resource overhead and an average global adaptation time of 75 ms

Embedded electronic systems driven by run-time reconfigurable hardware

Abstract This doctoral thesis addresses the design of embedded electronic systems based on run-time reconfigurable hardware technology –available through SRAM-based FPGA/SoC devices– aimed at contributing to enhance the life quality of the human beings. This work does research on the conception of the system architecture and the reconfiguration engine that provides to the FPGA the capability of dynamic partial reconfiguration in order to synthesize, by means of hardware/software co-design, a given application partitioned in processing tasks which are multiplexed in time and space, optimizing thus its physical implementation –silicon area, processing time, complexity, flexibility, functional density, cost and power consumption– in comparison with other alternatives based on static hardware (MCU, DSP, GPU, ASSP, ASIC, etc.). The design flow of such technology is evaluated through the prototyping of several engineering applications (control systems, mathematical coprocessors, complex image processors, etc.), showing a high enough level of maturity for its exploitation in the industry.Resumen Esta tesis doctoral abarca el diseño de sistemas electrónicos embebidos basados en tecnología hardware dinámicamente reconfigurable –disponible a través de dispositivos lógicos programables SRAM FPGA/SoC– que contribuyan a la mejora de la calidad de vida de la sociedad. Se investiga la arquitectura del sistema y del motor de reconfiguración que proporcione a la FPGA la capacidad de reconfiguración dinámica parcial de sus recursos programables, con objeto de sintetizar, mediante codiseño hardware/software, una determinada aplicación particionada en tareas multiplexadas en tiempo y en espacio, optimizando así su implementación física –área de silicio, tiempo de procesado, complejidad, flexibilidad, densidad funcional, coste y potencia disipada– comparada con otras alternativas basadas en hardware estático (MCU, DSP, GPU, ASSP, ASIC, etc.). Se evalúa el flujo de diseño de dicha tecnología a través del prototipado de varias aplicaciones de ingeniería (sistemas de control, coprocesadores aritméticos, procesadores de imagen, etc.), evidenciando un nivel de madurez viable ya para su explotación en la industria.Resum Aquesta tesi doctoral està orientada al disseny de sistemes electrònics empotrats basats en tecnologia hardware dinàmicament reconfigurable –disponible mitjançant dispositius lògics programables SRAM FPGA/SoC– que contribueixin a la millora de la qualitat de vida de la societat. S’investiga l’arquitectura del sistema i del motor de reconfiguració que proporcioni a la FPGA la capacitat de reconfiguració dinàmica parcial dels seus recursos programables, amb l’objectiu de sintetitzar, mitjançant codisseny hardware/software, una determinada aplicació particionada en tasques multiplexades en temps i en espai, optimizant així la seva implementació física –àrea de silici, temps de processat, complexitat, flexibilitat, densitat funcional, cost i potència dissipada– comparada amb altres alternatives basades en hardware estàtic (MCU, DSP, GPU, ASSP, ASIC, etc.). S’evalúa el fluxe de disseny d’aquesta tecnologia a través del prototipat de varies aplicacions d’enginyeria (sistemes de control, coprocessadors aritmètics, processadors d’imatge, etc.), demostrant un nivell de maduresa viable ja per a la seva explotació a la indústria

From MARTE to dynamically reconfigurable FPGAs : Introduction of a control extension in a model based design flow

System-on-Chip (SoC) can be considered as a particular case of embedded systems and has rapidly became a de-facto solution for implement- ing these complex systems. However, due to the continuous exponential rise in SoC's design complexity, there is a critical need to find new seamless method- ologies and tools to handle the SoC co-design aspects. This paper addresses this issue and proposes a novel SoC co-design methodology based on Model Driven Engineering (MDE) and the MARTE (Modeling and Analysis of Real-Time and Embedded Systems) standard proposed by OMG (Object Management Group), in order to raise the design abstraction levels. Extensions of this standard have enabled us to move from high level specifications to execution platforms such as reconfigurable FPGAs; and allow to implement the notion of Partial Dy- namic Reconfiguration supported by current FPGAs. The overall objective is to carry out system modeling at a high abstraction level expressed in UML (Unified Modeling Language); and afterwards, transform these high level mod- els into detailed enriched lower level models in order to automatically generate the necessary code for final FPGA synthesis

A dynamic partial reconfiguration design for camera systems

The image-processing pipeline is the core part of any camera system including digital still cameras, camcorders, camera phones and video surveillance equipments. The image-processing pipeline consists of a number of processing stages that enhance the image or remove any effects that are caused by surrounding conditions. These stages are computationally intensive and need special requirements to meet the real time processing. This paper discusses the pipeline parts and presents a high-performance and cost-effective implementation of the pipeline on Field Programmable Gate Arrays (FPGAs) using Dynamic Partial Reconfiguration (DPR) feature to exploit the FPGA resources over time and space. The paper shows that the implemented system adds much of flexibility to camera systems by using a reconfigurable region. The system can use an unlimited number of image processing pipeline stages to process the images without the need of huge number of logic resources to fit all the stages. Moreover, the stages are not fixed in this system, they can be changed upon the user's decision. The architecture is designed to process still images of size 1920×1080. Each stage could process a full frame within 7.25 ms. A fast configuration engine is designed and deployed in the system. The engine shows that it can outperform the engine provided with zynq SoC by three times. The overall throughput of the system reaches 250 Megapixel/s

Towards the development of flexible, reliable, reconfigurable, and high-performance imaging systems

Current FPGAs can implement large systems because of the high density of reconfigurable logic resources in a single chip. FPGAs are comprehensive devices that combine flexibility and high performance in the same platform compared to other platform such as General-Purpose Processors (GPPs) and Application Specific Integrated Circuits (ASICs). The flexibility of modern FPGAs is further enhanced by introducing Dynamic Partial Reconfiguration (DPR) feature, which allows for changing the functionality of part of the system while other parts are functioning. FPGAs became an important platform for digital image processing applications because of the aforementioned features. They can fulfil the need of efficient and flexible platforms that execute imaging tasks efficiently as well as the reliably with low power, high performance and high flexibility. The use of FPGAs as accelerators for image processing outperforms most of the current solutions. Current FPGA solutions can to load part of the imaging application that needs high computational power on dedicated reconfigurable hardware accelerators while other parts are working on the traditional solution to increase the system performance. Moreover, the use of the DPR feature enhances the flexibility of image processing further by swapping accelerators in and out at run-time. The use of fault mitigation techniques in FPGAs enables imaging applications to operate in harsh environments following the fact that FPGAs are sensitive to radiation and extreme conditions. The aim of this thesis is to present a platform for efficient implementations of imaging tasks. The research uses FPGAs as the key component of this platform and uses the concept of DPR to increase the performance, flexibility, to reduce the power dissipation and to expand the cycle of possible imaging applications. In this context, it proposes the use of FPGAs to accelerate the Image Processing Pipeline (IPP) stages, the core part of most imaging devices. The thesis has a number of novel concepts. The first novel concept is the use of FPGA hardware environment and DPR feature to increase the parallelism and achieve high flexibility. The concept also increases the performance and reduces the power consumption and area utilisation. Based on this concept, the following implementations are presented in this thesis: An implementation of Adams Hamilton Demosaicing algorithm for camera colour interpolation, which exploits the FPGA parallelism to outperform other equivalents. In addition, an implementation of Automatic White Balance (AWB), another IPP stage that employs DPR feature to prove the mentioned novelty aspects. Another novel concept in this thesis is presented in chapter 6, which uses DPR feature to develop a novel flexible imaging system that requires less logic and can be implemented in small FPGAs. The system can be employed as a template for any imaging application with no limitation. Moreover, discussed in this thesis is a novel reliable version of the imaging system that adopts novel techniques including scrubbing, Built-In Self Test (BIST), and Triple Modular Redundancy (TMR) to detect and correct errors using the Internal Configuration Access Port (ICAP) primitive. These techniques exploit the datapath-based nature of the implemented imaging system to improve the system's overall reliability. The thesis presents a proposal for integrating the imaging system with the Robust Reliable Reconfigurable Real-Time Heterogeneous Operating System (R4THOS) to get the best out of the system. The proposal shows the suitability of the proposed DPR imaging system to be used as part of the core system of autonomous cars because of its unbounded flexibility. These novel works are presented in a number of publications as shown in section 1.3 later in this thesis

Immune System Based Control and Intelligent Agent Design for Power System Applications

The National Academy of Engineering has selected the US Electric Power Grid as the supreme engineering achievement of the 20th century. Yet, this same grid is struggling to keep up with the increasing demand for electricity, its quality and cost. A growing recognition of the need to modernize the grid to meet future challenges has found articulation in the vision of a Smart Grid in using new control strategies that are intelligent, distributed, and adaptive. The objective of this work is to develop smart control systems inspired from the biological Human Immune System to better manage the power grid at the both generation and distribution levels. The work is divided into three main sections. In the first section, we addressed the problem of Automatic Generation Control design. The Clonal Selection theory is successfully applied as an optimization technique to obtain decentralized control gains that minimize a performance index based on Area Control Errors. Then the Immune Network theory is used to design adaptive controllers in order to diminish the excess maneuvering of the units and help the control areas comply with the North American Electric Reliability Corporation\u27s standards set to insure good quality of service and equitable mutual assistance by the interconnected energy balancing areas. The second section of this work addresses the design and deployment of Multi Agent Systems on both terrestrial and shipboard power systems self-healing using a novel approach based on the Immune Multi-Agent System (IMAS). The Immune System is viewed as a highly organized and distributed Multi-Cell System that strives to heal the body by working together and communicating to get rid of the pathogens. In this work both simulation and hardware design and deployment of the MAS are addressed. The third section of this work consists in developing a small scale smart circuit by modifying and upgrading the existing Analog Power Simulator to demonstrate the effectiveness of the developed technologies. We showed how to develop smart Agents hardware along with a wireless communication platform and the electronic switches. After putting together the different designed pieces, the resulting Multi Agent System is integrated into the Power Simulator Hardware. The multi Agent System developed is tested for fault isolation, reconfiguration, and restoration problems by simulating a permanent three phase fault on one of the feeder lines. The experimental results show that the Multi Agent System hardware developed performed effectively and in a timely manner which confirms that this technology is very promising and a very good candidate for Smart Grid control applications

TOWARDS A NOVEL RESILIENT ROBOTIC SYSTEM

Resilient robotic systems are a kind of robotic system that is able to recover their original function after partial damage of the system. This is achieved by making changes on the partially damaged robot. In this dissertation study, a general robot, which makes sense by including active joints, passive joints, passive links, and passive adjustable links, was proposed in order to explore its resilience. Note that such a robot is also called an under-actuated robot. This dissertation presents the following studies. First, a novel architecture of robots was proposed, which is characterized as under-actuated robot. The architecture enables three types of recovery strategy, namely (1) change of the robot behavior, (2) change of the robot state, and (3) change of the robot configuration. Second, a novel docking system was developed, which allows for the realization of real-time assembly and disassembly and passive joint and adjustable passive link, and this thus enables the realization of the proposed architecture. Third, an example prototype system was built to experiment the effectiveness of the proposed architecture and to demonstrate the resilient behavior of the robot. Fourth, a novel method for robot configuration synthesis was developed, which is based on the genetic algorithm (GA), to determine the goal configuration of a partially damaged robot, at which the robot can still perform its original function. The novelty of the method lies in the integration of both discrete variables such as the number of modules, type of modules, and assembly patterns between modules and the continuous variables such as the length of modules and initial location of the robot. Fifth, a GA-based method for robot reconfiguration planning and scheduling was developed to actually change the robot from its initial configuration to the goal configuration with a minimum effort (time and energy). Two conclusions can be drawn from the above studies. First, the under-actuated robotic architecture can build a cost effective robot that can achieve the highest degree of resilience. Second, the design of the under-actuated resilient robot with the proposed docking system not only reduces the cost but also overcomes the two common actuator failures: (i) an active joint is unlocked (thus becoming a passive joint) and (ii) an active joint is locked (thus becoming an adjustable link). There are several contributions made by this dissertation to the field of robotics. The first is the finding that an under-actuated robot can be made more resilient. In the field of robotics, the concept of the under-actuated robot is available, but it has not been considered for reconfiguration (in literature, the reconfiguration is mostly about fully actuated robots). The second is the elaboration on the concept of reconfiguration planning, scheduling, and manipulation/control. In the literature of robotics, only the concept of reconfiguration planning is precisely given but not for reconfiguration scheduling. The third is the development of the model along with its algorithm for synthesis of the goal reconfiguration, reconfiguration planning, and scheduling. The application of the proposed under-actuated resilient robot lies in the operations in unknown or dangerous environments, for example, in rescue missions and space explorations. In these applications, replacement or repair of a damaged robot is impossible or cost-prohibited

Energy challenges for ICT

The energy consumption from the expanding use of information and communications technology (ICT) is unsustainable with present drivers, and it will impact heavily on the future climate change. However, ICT devices have the potential to contribute signi - cantly to the reduction of CO2 emission and enhance resource e ciency in other sectors, e.g., transportation (through intelligent transportation and advanced driver assistance systems and self-driving vehicles), heating (through smart building control), and manu- facturing (through digital automation based on smart autonomous sensors). To address the energy sustainability of ICT and capture the full potential of ICT in resource e - ciency, a multidisciplinary ICT-energy community needs to be brought together cover- ing devices, microarchitectures, ultra large-scale integration (ULSI), high-performance computing (HPC), energy harvesting, energy storage, system design, embedded sys- tems, e cient electronics, static analysis, and computation. In this chapter, we introduce challenges and opportunities in this emerging eld and a common framework to strive towards energy-sustainable ICT