Application-aware optimization of Artificial Intelligence for deployment on resource constrained devices

Artificial intelligence (AI) is changing people's everyday life. AI techniques such as Deep Neural Networks (DNN) rely on heavy computational models, which are in principle designed to be executed on powerful HW platforms, such as desktop or server environments. However, the increasing need to apply such solutions in people's everyday life has encouraged the research for methods to allow their deployment on embedded, portable and stand-alone devices, such as mobile phones, which exhibit relatively low memory and computational resources. Such methods targets both the development of lightweight AI algorithms and their acceleration through dedicated HW. This thesis focuses on the development of lightweight AI solutions, with attention to deep neural networks, to facilitate their deployment on resource constrained devices. Focusing on the computer vision field, we show how putting together the self learning ability of deep neural networks with application-specific knowledge, in the form of feature engineering, it is possible to dramatically reduce the total memory and computational burden, thus allowing the deployment on edge devices. The proposed approach aims to be complementary to already existing application-independent network compression solutions. In this work three main DNN optimization goals have been considered: increasing speed and accuracy, allowing training at the edge, and allowing execution on a microcontroller. For each of these we deployed the resulting algorithm to the target embedded device and measured its performance

FPGA Implementation for Real-Time Background Subtraction Based on Horprasert Model

Background subtraction is considered the first processing stage in video surveillance systems, and consists of determining objects in movement in a scene captured by a static camera. It is an intensive task with a high computational cost. This work proposes an embedded novel architecture on FPGA which is able to extract the background on resource-limited environments and offers low degradation (produced because of the hardware-friendly model modification). In addition, the original model is extended in order to detect shadows and improve the quality of the segmentation of the moving objects. We have analyzed the resource consumption and performance in Spartan3 Xilinx FPGAs and compared to others works available on the literature, showing that the current architecture is a good trade-off in terms of accuracy, performance and resources utilization. With less than a 65% of the resources utilization of a XC3SD3400 Spartan-3A low-cost family FPGA, the system achieves a frequency of 66.5 MHz reaching 32.8 fps with resolution 1,024 × 1,024 pixels, and an estimated power consumption of 5.76 W

Cross-layer Optimized Wireless Video Surveillance

A wireless video surveillance system contains three major components, the video capture and preprocessing, the video compression and transmission over wireless sensor networks (WSNs), and the video analysis at the receiving end. The coordination of different components is important for improving the end-to-end video quality, especially under the communication resource constraint. Cross-layer control proves to be an efficient measure for optimal system configuration. In this dissertation, we address the problem of implementing cross-layer optimization in the wireless video surveillance system. The thesis work is based on three research projects. In the first project, a single PTU (pan-tilt-unit) camera is used for video object tracking. The problem studied is how to improve the quality of the received video by jointly considering the coding and transmission process. The cross-layer controller determines the optimal coding and transmission parameters, according to the dynamic channel condition and the transmission delay. Multiple error concealment strategies are developed utilizing the special property of the PTU camera motion. In the second project, the binocular PTU camera is adopted for video object tracking. The presented work studied the fast disparity estimation algorithm and the 3D video transcoding over the WSN for real-time applications. The disparity/depth information is estimated in a coarse-to-fine manner using both local and global methods. The transcoding is coordinated by the cross-layer controller based on the channel condition and the data rate constraint, in order to achieve the best view synthesis quality. The third project is applied for multi-camera motion capture in remote healthcare monitoring. The challenge is the resource allocation for multiple video sequences. The presented cross-layer design incorporates the delay sensitive, content-aware video coding and transmission, and the adaptive video coding and transmission to ensure the optimal and balanced quality for the multi-view videos. In these projects, interdisciplinary study is conducted to synergize the surveillance system under the cross-layer optimization framework. Experimental results demonstrate the efficiency of the proposed schemes. The challenges of cross-layer design in existing wireless video surveillance systems are also analyzed to enlighten the future work. Adviser: Song C

Cross-layer Optimized Wireless Video Surveillance

A wireless video surveillance system contains three major components, the video capture and preprocessing, the video compression and transmission over wireless sensor networks (WSNs), and the video analysis at the receiving end. The coordination of different components is important for improving the end-to-end video quality, especially under the communication resource constraint. Cross-layer control proves to be an efficient measure for optimal system configuration. In this dissertation, we address the problem of implementing cross-layer optimization in the wireless video surveillance system. The thesis work is based on three research projects. In the first project, a single PTU (pan-tilt-unit) camera is used for video object tracking. The problem studied is how to improve the quality of the received video by jointly considering the coding and transmission process. The cross-layer controller determines the optimal coding and transmission parameters, according to the dynamic channel condition and the transmission delay. Multiple error concealment strategies are developed utilizing the special property of the PTU camera motion. In the second project, the binocular PTU camera is adopted for video object tracking. The presented work studied the fast disparity estimation algorithm and the 3D video transcoding over the WSN for real-time applications. The disparity/depth information is estimated in a coarse-to-fine manner using both local and global methods. The transcoding is coordinated by the cross-layer controller based on the channel condition and the data rate constraint, in order to achieve the best view synthesis quality. The third project is applied for multi-camera motion capture in remote healthcare monitoring. The challenge is the resource allocation for multiple video sequences. The presented cross-layer design incorporates the delay sensitive, content-aware video coding and transmission, and the adaptive video coding and transmission to ensure the optimal and balanced quality for the multi-view videos. In these projects, interdisciplinary study is conducted to synergize the surveillance system under the cross-layer optimization framework. Experimental results demonstrate the efficiency of the proposed schemes. The challenges of cross-layer design in existing wireless video surveillance systems are also analyzed to enlighten the future work. Adviser: Song C

Architectures and Algorithms for the Signal Processing of Advanced MIMO Radar Systems

This thesis focuses on the research, development and implementation of novel concepts, architectures, demonstrator systems and algorithms for the signal processing of advanced Multiple Input Multiple Output (MIMO) radar systems. The key concept is to address compact system, which have high resolutions and are able to perform a fast radar signal processing, three-dimensional (3D), and four-dimensional (4D) beamforming for radar image generation and target estimation. The idea is to obtain a complete sensing of range, Azimuth and elevation (additionally Doppler as the fourth dimension) from the targets in the radar captures. The radar technology investigated, aims at addressing sev- eral civil and military applications, such as surveillance and detection of targets, both air and ground based, and situational awareness, both in cars and in flying platforms, from helicopters, to Unmanned Aerial Vehicles (UAV) and air-taxis. Several major topics have been targeted. The development of complete systems and innovative FPGA, ARM and software based digital architectures for 3D imaging MIMO radars, which operate in both Time Division Multiplexing (TDM) and Frequency Divi- sion Multiplexing (FDM) modes, with Frequency Modulated Continuous Wave (FMCW) and Orthogonal Frequency Division Multiplexing (OFDM) signals, respectively. The de- velopment of real-time radar signal processing, beamforming and Direction-Of-Arrival (DOA) algorithms for target detection, with particular focus on FFT based, hardware implementable techniques. The study and implementation of advanced system concepts, parametrisation and simulation of next generation real-time digital radars (e.g. OFDM based). The design and development of novel constant envelope orthogonal waveforms for real-time 3D OFDM MIMO radar systems. The MIMO architectures presented in this thesis are a collection of system concepts, de- sign and simulations, as well as complete radar demonstrators systems, with indoor and outdoor measurements. Several of the results shown, come in the form of radar images which have been captured in field-test, in different scenarios, which aid in showing the proper functionality of the systems. The research activities for this thesis, have been carried out on the premises of Air- bus, based in Munich (Germany), as part of a Ph.D. candidate joint program between Airbus and the Polytechnic Department of Engineering and Architecture (Dipartimento Politecnico di Ingegneria e Architettura), of the University of Udine, based in Udine (Italy).Questa tesi si concentra sulla ricerca, lo sviluppo e l\u2019implementazione di nuovi concetti, architetture, sistemi dimostrativi e algoritmi per l\u2019elaborazione dei segnali in sistemi radar avanzati, basati su tecnologia Multiple Input Multiple Output (MIMO). Il con- cetto chiave `e quello di ottenere sistemi compatti, dalle elevate risoluzioni e in grado di eseguire un\u2019elaborazione del segnale radar veloce, un beam-forming tri-dimensionale (3D) e quadri-dimensionale (4D) per la generazione di immagini radar e la stima delle informazioni dei bersagli, detti target. L\u2019idea `e di ottenere una stima completa, che includa la distanza, l\u2019Azimuth e l\u2019elevazione (addizionalmente Doppler come quarta di- mensione) dai target nelle acquisizioni radar. La tecnologia radar indagata ha lo scopo di affrontare diverse applicazioni civili e militari, come la sorveglianza e la rilevazione di targets, sia a livello aereo che a terra, e la consapevolezza situazionale, sia nelle auto che nelle piattaforme di volo, dagli elicotteri, ai Unmanned Aerial Vehicels (UAV) e taxi volanti (air-taxis). Le tematiche affrontante sono molte. Lo sviluppo di sistemi completi e di architetture digitali innovative, basate su tecnologia FPGA, ARM e software, per radar 3D MIMO, che operano in modalit`a Multiplexing Time Division Multiplexing (TDM) e Multiplexing Frequency Diversion (FDM), con segnali di tipo FMCW (Frequency Modulated Contin- uous Wave) e Orthogonal Frequency Division Multiplexing (OFDM), rispettivamente. Lo sviluppo di tecniche di elaborazione del segnale radar in tempo reale, algoritmi di beam-forming e di stima della direzione di arrivo, Direction-Of-Arrival (DOA), dei seg- nali radar, per il rilevamento dei target, con particolare attenzione a processi basati su trasformate di Fourier (FFT). Lo studio e l\u2019implementazione di concetti di sistema avan- zati, parametrizzazione e simulazione di radar digitali di prossima generazione, capaci di operare in tempo reale (ad esempio basati su architetture OFDM). Progettazione e sviluppo di nuove forme d\u2019onda ortogonali ad inviluppo costante per sistemi radar 3D di tipo OFDM MIMO, operanti in tempo reale. Le attivit`a di ricerca di questa tesi sono state svolte presso la compagnia Airbus, con sede a Monaco di Baviera (Germania), nell\u2019ambito di un programma di dottorato, svoltosi in maniera congiunta tra Airbus ed il Dipartimento Politecnico di Ingegneria e Architettura dell\u2019Universit`a di Udine, con sede a Udine

High Dynamic Range Adaptive Real-time Smart Camera: an overview of the HDR-ARTiST project

International audienceStandard cameras capture only a fraction of the information that is visible to the human visual system. This is specifically true for natural scenes including areas of low and high illumination due to transitions between sunlit and shaded areas. When capturing such a scene, many cameras are unable to store the full Dynamic Range (DR) resulting in low quality video where details are concealed in shadows or washed out by sunlight. The imaging technique that can overcome this problem is called HDR (High Dynamic Range) imaging. This paper describes a complete smart camera built around a standard off-the-shelf LDR (Low Dynamic Range) sensor and a Virtex-6 FPGA board. This smart camera called HDR-ARtiSt (High Dynamic Range Adaptive Real-time Smart camera) is able to produce a real-time HDR live video color stream by recording and combining multiple acquisitions of the same scene while varying the exposure time. This technique appears as one of the most appropriate and cheapest solution to enhance the dynamic range of real-life environments. HDR-ARtiSt embeds real-time multiple captures, HDR processing, data display and transfer of a HDR color video for a full sensor resolution (1280 1024 pixels) at 60 frames per second. The main contributions of this work are: (1) Multiple Exposure Control (MEC) dedicated to the smart image capture with alternating three exposure times that are dynamically evaluated from frame to frame, (2) Multi-streaming Memory Management Unit (MMMU) dedicated to the memory read/write operations of the three parallel video streams, corresponding to the different exposure times, (3) HRD creating by combining the video streams using a specific hardware version of the Devebecs technique, and (4) Global Tone Mapping (GTM) of the HDR scene for display on a standard LCD monitor