A New Development Framework for Multi-Core Processor based Smart-Camera Implementations

International audienceThe exponential evolution of the smart camera processing performances is directly linked to the improvements on hardware processing elements. Nowadays, high processing performances can be reached considering hardware targets which enables a high level of task parallelism to be implemented. Highly regular tasks are good candidate for a reconfigurable logic implementation and less regular parts of the algorithm could be described on the processor. Meanwhile the prototyping time is related to the selected target and the associated development methodology. The implementation on reconfigurable logic is highly efficient in exploiting the intrinsic task parallelism nevertheless can be time consuming using traditional methodology (i.e. Hardware Language Description). Several approaches can be considered to decrease the proto-typing time and to conserve high processing performances for instance implementation based on: • heterogeneous architectures [1] that mixed reconfig-urable logic (i.e. FPGA) and embedded processor, • high-level abstraction description and the associated fast prototyping tools [2][3][4], • multi-core processor architectures such as Digital Signal Processors (DSP), Graphic Processor Units (GPU) or even Generic Purpose Processor (GPP). In this paper, we propose to focus on implementation based on GPP due to the emergence of new generation of low-cost multi-core processors which enables high processing performances to be reached and therefore to match with some constraints of complex image-processing algorithms. The key idea of this development is to be able to propose fast prototyping using a low-cost smart camera based on this kind of target. Hence, we have developed a new framework dedicated to multi-core processor associated with an image sensor. The framework aims to offer a high degree of flexibility for managing the tasks and the memory allocation. Hence, the framework enables the priority and the allocation of each task to be controlled. Each task (or binary) is independent in terms of execution nevertheless it can be linked and controlled using a higher hierarchy level binary. The image acquisition task can be completely independent from the other processing tasks. One processor's core can even be dedicated to the acquisition task to guarantee a constant input data-flow to the image processing tasks. The data exchange is defined in POSIX, each binary can be therefore coded differently (for instance in C or C++, or in another languages) and offer a relative Operating System (OS) compatibility. The memory management enables a sequence of images to be automatically stored and a simultaneous access to be granted for several processings. The framework includes an interface dedicated to the management of the tasks: the user can add or suppress a binary during the runtime, logs or processing results can be visualised for each task

WiseEye: A Platform to Manage and Experiment on Smart Camera Networks

International audienceEmbedded vision is probably at the edge of phenomenal expansion. The smart cameras are embedding some processing units which are more and more powerful. Last decade, high-speed image processing can be implemented on specifically designed architectures [1] nevertheless the designing time of such systems was quite high and time to market therefore as well. Since, powerful chips (i.e System On Chip) and quick prototyping methodologies are contently emerging [2],[3],[4] and enable more complex algorithms to be implemented faster. Moreover, smart cameras which are embedding flexible and powerful multi-core processors or Graphic Processors Unit (GPU) are now available and can be considered as well as a solution to implement faster some complex image processing algorithms. The smart camera can be considered as a powerful sensor which enables very complex information to be extracted in real-time from the video scene. Using several cameras simultaneously and dealing with a multi-view configuration is even more challenging but enable more information to be available. Therefore, we present in this paper a platform, named WiseEye, to manage and experiment on a smart camera network based on low-cost multi-core processors. A network of low-cost multi-core processors has been deployed. We have already developed a framework to ease application development and debugging [5]. The framework aims to offer a high degree of flexibility for managing the tasks and the memory allocation. Hence, the framework enables the priority and the allocation of each task to be controlled. The image acquisition task can be completely independent from the other processing tasks. The framework includes an interface dedicated to the management of the tasks: the user can add or suppress a task during the runtime, logs or processing results can be visualised for each task. Smart cameras use a dedicated network configuration and service providing tool named pyM2SL (python Mesh and Multicast Services for Linux) which has been developed in le2i. pyM2SL allows dynamic application management and configuration from a master node on the network, and service deployment and discovery at runtime. Services can be anything from video stream to processing resources, each allocable according to the user's requirements. pyM2SL is a free software, publicly available [6] under AGPL v3 open source license. The viewer application has been designed using multi-platform libraries only, it is based on a Qt GUI. The received video streams can be decoded with a software decoder or with hardware to reduce the CPU usage. The video streaming is based on GStreamer which offers high performances (TCP or UDP protocols available) with very low latencies. We are currently investigating on the real-time implementation of complex image processing on this kind of hardware targets to provide extra services and security to the people living in an environment equipped with a smart camera network. Different targeted applications are then finally presented such as human tracking [7] for smart building management, control access based on a multi-modal approach, real-time fall detection [4] or recent remote physiological measurements (i.e. heart rate) based on video imaging [8]

Toward the evelopment of a remote photopletysmographic sensor

La mesure cardiaque sans contact réalisée grâce aux méthodes de photopléthysmographie sans contact est un domaine de recherche très actif. Depuis l'introduction en 2010 d'une nouvelle méthode de mesure avec des capteurs optiques d'entrée de gamme (webcam PC), les travaux de recherche se sont multipliés. Ainsi, on observe une plus grande diversité des méthodes proposées afin de réaliser la mesure. Egalement, la précision de la mesure a grandement progressé et les scénarios et possibilités d'usage de la technologie sont aujourd'hui très nombreux. Au cœur de ce processus de mesure, la segmentation dans l'image de la ou les zones d'intérêt est une étape clé. Nous proposons dans cette thèse une méthode innovante afin de réaliser la mesure photoplethysmographique sans contact en identifiant implicitement les zones de peau vivante dans la vidéo. Nous avons montré que notre approche permet d'améliorer la qualité de la mesure en favorisant les zones dans l'image où le signal est de plus grande qualité. Afin de rendre possible l'intégration de notre solution, nous avons proposé une nouvelle méthode de segmentation en superpixels, nommée IBIS, qui permet de réduire la complexité algorithmique de cette étape du traitement. Ce faisant, nous avons démontré la faisabilité de l'intégration de notre solution au sein d'une plateforme embarquée. Les différentes méthodes ont été évaluées au travers de plusieurs expérimentations afin de valider leurs performances. Notre méthode de segmentation en superpixels est comparée aux méthodes de l'état de l'art tandis que nous avons implémenté plusieurs des méthodes de mesure du signal photoplethysmographique afin de discuter de l'impact de notre approche sur la qualité de la mesure photoplethysmographique. Que ce soit pour la segmentation en superpixels ou pour l'estimation du rythme cardiaque sans contact, nous avons montré une importante plus-value de nos méthodes comparées à celles disponibles dans la littérature. Les différents travaux présentés dans ce document ont été valorisés au travers de publications en conférences et revue.Heart-rate estimation performed with remote photoplethysmography is a very active research field. Since pioneer works in 2010, which demonstrated the feasibility of the measure with low-grade consumers’ camera (webcam), the number of scientific publications have increased significantly in the domain. Hence, we observe a multiplication of the methods in order to retrieve the photoplethysmographic signal which has led to an increased precision and quality of the heart-rate estimation. Region of interest segmentation is a key step of the processing pipeline in order to maximize the quality of the measured signal. We propose a new method to perform remote photoplethysmographic measurement using an implicit living skin identification method. Hence, we have shown that our approach lead to an improvement in both quality of the signal measured and precision of the heart-rate estimation by favoring more contributive area. As we are working with hardware integration constraint, we propose a new superpixels segmentation method which requires significantly less computation power than state of the art methods by reducing the algorithmic complexity of this step. Moreover, we have demonstrated the integration and real time capabilities by implementing our solution to an embedded device. All of our proposed method have been evaluated through different experimentations. Our new segmentation method, called IBIS, have been compared to state of the art methods to quantify the quality of the produced segmentation. To quantify the impact of our approach on the quality of the photoplethysmographic measure, we have implemented and compared state of the art methods with our proposed method. For both the superpixels segmentation and remote heart-rate estimation, our methods have shown great results and advantages compared to state of the art ones. Our works have been reviewed by the scientific community through several conference presentations and journal publications

Vers le développement d'un capteur photopléthysmographique sans contact

Heart-rate estimation performed with remote photoplethysmography is a very active research field. Since pioneer works in 2010, which demonstrated the feasibility of the measure with low-grade consumers’ camera (webcam), the number of scientific publications have increased significantly in the domain. Hence, we observe a multiplication of the methods in order to retrieve the photoplethysmographic signal which hasled to an increased precision and quality of the heartrate estimation. Region of interest segmentation is a key step of the processing pipeline in order to maximize the quality of the measured signal. We propose a new method to perform remote photoplethysmographic measurement using an implicit living skin identification method. Hence, we have shown that our approach lead to an improvement in both quality of the signalmeasured and precision of the heart-rate estimation by favoring more contributive area. As we are working with hardware integration constraint, we propose a new superpixels segmentation method which requires significantly less computation power than state of the art methods by reducing the algorithmic complexity of this step. Moreover, we have demonstrated the integration and real time capabilities by implementing our solution to anembedded device. All of our proposed method have been evaluated through dierent experimentations. Our new segmentation method, called IBIS, have been compared to state of the art methods to quantify the quality of the produced segmentation. To quantify the impact of ourapproach on the quality of the photoplethysmographic measure, we have implemented and compared state of the art methods with our proposed method. For both the superpixels segmentation and remote heart-rate estimation, our methods have shown great results and advantages compared to state of the art ones. Our works have been reviewed by the scientific community through several conference presentations and journal publications.La mesure cardiaque sans contact réalisée grâce aux méthodes de photopléthysmographie sans contact est un domaine de recherche très actif. Depuis l’introduction en 2010 d’une nouvelle méthode de mesure avec des capteurs optiques d’entrée de gamme (webcam PC), les travaux de recherche se sont multipliés. Ainsi, on observe une plus grande diversité des méthodes proposées afin de réaliser la mesure. Egalement, la précision de la mesure a grandement progressé et les scénarios et possibilités d’usage de la technologie sont aujourd’hui très nombreux. Au cœur de ce processus de mesure, la segmentation dans l’image de la ou les zones d’intérêt est une étape clé. Nous proposons dans cette thèse une méthode innovante afin de réaliser la mesure photopléthysmographique sans contact en identifiant implicitement les zones de peau vivante dans la vidéo. Nous avons montré que notre approche permet d’améliorer la qualité de la mesure en favorisant les zones dans l’image où le signal est de plus grande qualité. Afin de rendre possible l’intégration de notre solution, nous avons propos´e une nouvelle méthode de segmentation en superpixels, nommée IBIS, qui permet de réduire la complexité algorithmique de cette étape du traitement. Ce faisant, nous avons démontré la faisabilité de l’intégration de notre solution au sein d’une plateforme embarquée. Les différentes méthodes ont été évaluées au travers de plusieurs expérimentations afin de valider leurs performances. Notre méthode de segmentation en superpixels est comparée aux méthodes de l’état de l’arttandis que nous avons implémenté plusieurs des méthodes de mesure du signal photopléthysmographique afin de discuter de l’impact de notre approche sur la qualité de la mesure photopléthysmographique. Que ce soit pour la segmentation en superpixels ou pour l’estimation du rythme cardiaque sans contact, nous avons montré une importante plus-value de nos méthodes comparées à celles disponibles dans la littérature. Les différents travaux présentés dans ce document ont été valorisés au travers de publications en conférences et revue

Vers le développement d'un capteur photoplétysmographique sans contact

Heart-rate estimation performed with remote photoplethysmography is a very active research field. Since pioneer works in 2010, which demonstrated the feasibility of the measure with low-grade consumers’ camera (webcam), the number of scientific publications have increased significantly in the domain. Hence, we observe a multiplication of the methods in order to retrieve the photoplethysmographic signal which has led to an increased precision and quality of the heart-rate estimation. Region of interest segmentation is a key step of the processing pipeline in order to maximize the quality of the measured signal. We propose a new method to perform remote photoplethysmographic measurement using an implicit living skin identification method. Hence, we have shown that our approach lead to an improvement in both quality of the signal measured and precision of the heart-rate estimation by favoring more contributive area. As we are working with hardware integration constraint, we propose a new superpixels segmentation method which requires significantly less computation power than state of the art methods by reducing the algorithmic complexity of this step. Moreover, we have demonstrated the integration and real time capabilities by implementing our solution to an embedded device. All of our proposed method have been evaluated through different experimentations. Our new segmentation method, called IBIS, have been compared to state of the art methods to quantify the quality of the produced segmentation. To quantify the impact of our approach on the quality of the photoplethysmographic measure, we have implemented and compared state of the art methods with our proposed method. For both the superpixels segmentation and remote heart-rate estimation, our methods have shown great results and advantages compared to state of the art ones. Our works have been reviewed by the scientific community through several conference presentations and journal publications.La mesure cardiaque sans contact réalisée grâce aux méthodes de photopléthysmographie sans contact est un domaine de recherche très actif. Depuis l'introduction en 2010 d'une nouvelle méthode de mesure avec des capteurs optiques d'entrée de gamme (webcam PC), les travaux de recherche se sont multipliés. Ainsi, on observe une plus grande diversité des méthodes proposées afin de réaliser la mesure. Egalement, la précision de la mesure a grandement progressé et les scénarios et possibilités d'usage de la technologie sont aujourd'hui très nombreux. Au cœur de ce processus de mesure, la segmentation dans l'image de la ou les zones d'intérêt est une étape clé. Nous proposons dans cette thèse une méthode innovante afin de réaliser la mesure photoplethysmographique sans contact en identifiant implicitement les zones de peau vivante dans la vidéo. Nous avons montré que notre approche permet d'améliorer la qualité de la mesure en favorisant les zones dans l'image où le signal est de plus grande qualité. Afin de rendre possible l'intégration de notre solution, nous avons proposé une nouvelle méthode de segmentation en superpixels, nommée IBIS, qui permet de réduire la complexité algorithmique de cette étape du traitement. Ce faisant, nous avons démontré la faisabilité de l'intégration de notre solution au sein d'une plateforme embarquée. Les différentes méthodes ont été évaluées au travers de plusieurs expérimentations afin de valider leurs performances. Notre méthode de segmentation en superpixels est comparée aux méthodes de l'état de l'art tandis que nous avons implémenté plusieurs des méthodes de mesure du signal photoplethysmographique afin de discuter de l'impact de notre approche sur la qualité de la mesure photoplethysmographique. Que ce soit pour la segmentation en superpixels ou pour l'estimation du rythme cardiaque sans contact, nous avons montré une importante plus-value de nos méthodes comparées à celles disponibles dans la littérature. Les différents travaux présentés dans ce document ont été valorisés au travers de publications en conférences et revue

Remote Photoplethysmography Based on Implicit Living Skin Tissue Segmentation

International audienceRegion of interest selection is an essential part for remote photoplethysmography (rPPG) algorithms. Most of the time, face detection provided by a supervised learning of physical appearance features coupled with skin detection is used for region of interest selection. However, both methods have several limitations and we propose to implicitly select living skin tissue via their particular pulsatility feature. The input video stream is decomposed into several temporal superpixels from which pulse signals are extracted. Pulsatility measure for each temporal superpixel is then used to merge pulse traces and estimate the photoplethysmogram signal. This allows to select skin tissue and furthermore to favor areas where the pulse trace is more predominant. Experimental results showed that our method perform better than state of the art algorithms without any critical face or skin detection

Periodic Variance Maximization using Generalized Eigenvalue Decomposition applied to Remote Photoplethysmography estimation

International audienceA generic periodic variance maximization algorithm to extract periodic or quasi-periodic signals of unknown periods embedded into multi-channel temporal signal recordings is described in this paper. The algorithm combines the notion of maximizing a periodicity metric combined with the global optimization scheme to estimate the source periodic signal of an unknown period. The periodicity maximization is performed using Generalized Eigenvalue Decomposition (GEVD) and the global optimization is performed using tabu search. A case study of remote photoplethysmography signal estimation has been utilized to assess the performance of the method using videos from public databases UBFC-RPPG [1] and MMSE-HR [31]. The results confirm the improved performance over existing state of the art methods and the feasibility of the use of the method in a live scenario owing to its small execution time

Unsupervised skin tissue segmentation for remote photoplethysmography

International audienceSegmentation is a critical step for many algorithms, especially for remote photoplethysmography (rPPG) applications as only the skin surface provides information. Moreover, it has been shown that the rPPG signal is not distributed homogeneously across the skin. Most of the time, algorithms get input information from face detection provided by a supervised learning of physical appearance and skin pixel selection. However, both methods show several limitations. In this paper, we propose a simple approach to implicitly select skin tissues based on their distinct pulsatility feature. The input video frames are decomposed into several temporal superpixels from which the pulse signals are extracted. A pulsatility measure from each temporal superpixel is then used to merge the pulse traces and estimate the photoplethysmogram signal. Since the most pulsatile signals provide high quality information, areas where the information is predominant are favored. We evaluated our contribution using a new publicly available dataset dedicated to rPPG algorithms comparison. The results of our experiments show that our method outperforms state of the art algorithms, without any critical face or skin detectio

Iterative Boundaries implicit Identification for superpixels Segmentation: a real-time approach

International audienc

Real-Time Temporal Superpixels for Unsupervised Remote Photoplethysmography

International audienceSegmentation is a critical step for many computer vision applications. Among them, the remote photoplethys-mography technique is significantly impacted by the quality of region of interest segmentation. With the heart-rate estimation accuracy, the processing time is obviously a key issue for real-time monitoring. Recent face detection algorithms can perform real-time processing, however for unsupervised algorithms, i.e. without any subject detection based on supervised learning, existing methods are not able to achieve real-time on regular platform. In this paper, we propose a new method to perform real-time un-supervised remote photoplethysmograhy based on efficient temporally propagated superpixels segmentation. The proposed method performs the segmentation step by implicitly identifying the superpixel boundaries. Hence, only a fraction of the image is used to perform the segmentation which reduces greatly the computational burden of the process. The segmentation quality remains comparable to state of the art methods while computational time is divided by a factor up to 8 times. The efficiency of the superpixel segmentation allow us to propose a real-time unsupervised rPPG algorithm considering frames of 640x480, RGB, at 25 frames per second on a single core platform. We obtained real-time processing for 93% of precision at 2.5 beat per minute using our inhouse video database