WATCHING PEOPLE: ALGORITHMS TO STUDY HUMAN MOTION AND ACTIVITIES

Nowadays human motion analysis is one of the most active research topics in Computer Vision and it is receiving an increasing attention from both the industrial and scientific communities. The growing interest in human motion analysis is motivated by the increasing number of promising applications, ranging from surveillance, human–computer interaction, virtual reality to healthcare, sports, computer games and video conferencing, just to name a few. The aim of this thesis is to give an overview of the various tasks involved in visual motion analysis of the human body and to present the issues and possible solutions related to it. In this thesis, visual motion analysis is categorized into three major areas related to the interpretation of human motion: tracking of human motion using virtual pan-tilt-zoom (vPTZ) camera, recognition of human motions and human behaviors segmentation. In the field of human motion tracking, a virtual environment for PTZ cameras (vPTZ) is presented to overcame the mechanical limitations of PTZ cameras. The vPTZ is built on equirectangular images acquired by 360° cameras and it allows not only the development of pedestrian tracking algorithms but also the comparison of their performances. On the basis of this virtual environment, three novel pedestrian tracking algorithms for 360° cameras were developed, two of which adopt a tracking-by-detection approach while the last adopts a Bayesian approach. The action recognition problem is addressed by an algorithm that represents actions in terms of multinomial distributions of frequent sequential patterns of different length. Frequent sequential patterns are series of data descriptors that occur many times in the data. The proposed method learns a codebook of frequent sequential patterns by means of an apriori-like algorithm. An action is then represented with a Bag-of-Frequent-Sequential-Patterns approach. In the last part of this thesis a methodology to semi-automatically annotate behavioral data given a small set of manually annotated data is presented. The resulting methodology is not only effective in the semi-automated annotation task but can also be used in presence of abnormal behaviors, as demonstrated empirically by testing the system on data collected from children affected by neuro-developmental disorders

Real-time human action and gesture recognition using skeleton joints information towards medical applications

Des efforts importants ont été faits pour améliorer la précision de la détection des actions humaines à l’aide des articulations du squelette. Déterminer les actions dans un environnement bruyant reste une tâche difficile, car les coordonnées cartésiennes des articulations du squelette fournies par la caméra de détection à profondeur dépendent de la position de la caméra et de la position du squelette. Dans certaines applications d’interaction homme-machine, la position du squelette et la position de la caméra ne cessent de changer. La méthode proposée recommande d’utiliser des valeurs de position relatives plutôt que des valeurs de coordonnées cartésiennes réelles. Les récents progrès des réseaux de neurones à convolution (RNC) nous aident à obtenir une plus grande précision de prédiction en utilisant des entrées sous forme d’images. Pour représenter les articulations du squelette sous forme d’image, nous devons représenter les informations du squelette sous forme de matrice avec une hauteur et une largeur égale. Le nombre d’articulations du squelette fournit par certaines caméras de détection à profondeur est limité, et nous devons dépendre des valeurs de position relatives pour avoir une représentation matricielle des articulations du squelette. Avec la nouvelle représentation des articulations du squelette et le jeu de données MSR, nous pouvons obtenir des performances semblables à celles de l’état de l’art. Nous avons utilisé le décalage d’image au lieu de l’interpolation entre les images, ce qui nous aide également à obtenir des performances similaires à celle de l’état de l’art.There have been significant efforts in the direction of improving accuracy in detecting human action using skeleton joints. Recognizing human activities in a noisy environment is still challenging since the cartesian coordinate of the skeleton joints provided by depth camera depends on camera position and skeleton position. In a few of the human-computer interaction applications, skeleton position, and camera position keep changing. The proposed method recommends using relative positional values instead of actual cartesian coordinate values. Recent advancements in CNN help us to achieve higher prediction accuracy using input in image format. To represent skeleton joints in image format, we need to represent skeleton information in matrix form with equal height and width. With some depth cameras, the number of skeleton joints provided is limited, and we need to depend on relative positional values to have a matrix representation of skeleton joints. We can show the state-of-the-art prediction accuracy on MSR data with the help of the new representation of skeleton joints. We have used frames shifting instead of interpolation between frames, which helps us achieve state-of-the-art performance

Investigation of Computer Vision Concepts and Methods for Structural Health Monitoring and Identification Applications

This study presents a comprehensive investigation of methods and technologies for developing a computer vision-based framework for Structural Health Monitoring (SHM) and Structural Identification (St-Id) for civil infrastructure systems, with particular emphasis on various types of bridges. SHM is implemented on various structures over the last two decades, yet, there are some issues such as considerable cost, field implementation time and excessive labor needs for the instrumentation of sensors, cable wiring work and possible interruptions during implementation. These issues make it only viable when major investments for SHM are warranted for decision making. For other cases, there needs to be a practical and effective solution, which computer-vision based framework can be a viable alternative. Computer vision based SHM has been explored over the last decade. Unlike most of the vision-based structural identification studies and practices, which focus either on structural input (vehicle location) estimation or on structural output (structural displacement and strain responses) estimation, the proposed framework combines the vision-based structural input and the structural output from non-contact sensors to overcome the limitations given above. First, this study develops a series of computer vision-based displacement measurement methods for structural response (structural output) monitoring which can be applied to different infrastructures such as grandstands, stadiums, towers, footbridges, small/medium span concrete bridges, railway bridges, and long span bridges, and under different loading cases such as human crowd, pedestrians, wind, vehicle, etc. Structural behavior, modal properties, load carrying capacities, structural serviceability and performance are investigated using vision-based methods and validated by comparing with conventional SHM approaches. In this study, some of the most famous landmark structures such as long span bridges are utilized as case studies. This study also investigated the serviceability status of structures by using computer vision-based methods. Subsequently, issues and considerations for computer vision-based measurement in field application are discussed and recommendations are provided for better results. This study also proposes a robust vision-based method for displacement measurement using spatio-temporal context learning and Taylor approximation to overcome the difficulties of vision-based monitoring under adverse environmental factors such as fog and illumination change. In addition, it is shown that the external load distribution on structures (structural input) can be estimated by using visual tracking, and afterward load rating of a bridge can be determined by using the load distribution factors extracted from computer vision-based methods. By combining the structural input and output results, the unit influence line (UIL) of structures are extracted during daily traffic just using cameras from which the external loads can be estimated by using just cameras and extracted UIL. Finally, the condition assessment at global structural level can be achieved using the structural input and output, both obtained from computer vision approaches, would give a normalized response irrespective of the type and/or load configurations of the vehicles or human loads

Deformability-induced effects of red blood cells in flow

To ensure a proper health state in the human body, a steady transport of blood is necessary. As the main cellular constituent in the blood suspension, red blood cells (RBCs) are governing the physical properties of the entire blood flow. Remarkably, these RBCs can adapt their shape to the prevailing surrounding flow conditions, ultimately allowing them to pass through narrow capillaries smaller than their equilibrium diameter. However, several diseases such as diabetes mellitus or malaria are linked to an alteration of the deformability. In this work, we investigate the shapes of RBCs in microcapillary flow in vitro, culminating in a shape phase diagram of two distinct, hydrodynamically induced shapes, the croissant and the slipper. Due to the simplicity of the RBC structure, the obtained phase diagram leads to further insights into the complex interaction between deformable objects in general, such as vesicles, and the surrounding fluid. Furthermore, the phase diagram is highly correlated to the deformability of the RBCs and represents thus a cornerstone of a potential diagnostic tool to detect pathological blood parameters. To further promote this idea, we train a convolutional neural network (CNN) to classify the distinct RBC shapes. The benchmark of the CNN is validated by manual classification of the cellular shapes and yields very good performance. In the second part, we investigate an effect that is associated with the deformability of RBCs, the lingering phenomenon. Lingering events may occur at bifurcation apices and are characterized by a straddling of RBCs at an apex, which have been shown in silico to cause a piling up of subsequent RBCs. Here, we provide insight into the dynamics of such lingering events in vivo, which we consequently relate to the partitioning of RBCs at bifurcating vessels in the microvasculature. Specifically, the lingering of RBCs causes an increased intercellular distance to RBCs further downstream, and thus, a reduced hematocrit.Um die biologischen Funktionen im menschlichen Körper aufrechtzuerhalten ist eine stetige Versorgung mit Blut notwendig. Rote Blutzellen bilden den Hauptanteil aller zellulären Komponenten im Blut und beeinflussen somit maßgeblich dessen Fließeigenschaften. Eine bemerkenswerte Eigenschaft dieser roten Blutzellen ist ihre Deformierbarkeit, die es ihnen ermöglicht, ihre Form den vorherrschenden Strömungsbedingungen anzupassen und sogar durch Kapillaren zu strömen, deren Durchmesser kleiner ist als der Gleichgewichtsdurchmesser einer roten Blutzelle. Zahlreiche Erkrankungen wie beispielsweise Diabetes mellitus oder Malaria sind jedoch mit einer Veränderung dieser Deformierbarkeit verbunden. In der vorliegenden Arbeit untersuchen wir die hydrodynamisch induzierten Formen der roten Blutzellen in mikrokapillarer Strömung in vitro systematisch für verschiedene Fließgeschwindigkeiten. Aus diesen Daten erzeugen wir ein Phasendiagramm zweier charakteristischer auftretender Formen: dem Croissant und dem Slipper. Aufgrund der Einfachheit der Struktur der roten Blutzellen führt das erhaltene Phasendiagramm zu weiteren Erkenntnissen über die komplexe Interaktion zwischen deformierbaren Objekten im Allgemeinen, wie z.B. Vesikeln, und des sie umgebenden Fluids. Darüber hinaus ist das Phasendiagramm korreliert mit der Deformierbarkeit der Erythrozyten und stellt somit einen Eckpfeiler eines potentiellen Diagnosewerkzeugs zur Erkennung pathologischer Blutparameter dar. Um diese Idee weiter voranzutreiben, trainieren wir ein künstliches neuronales Netz, um die auftretenden Formen der Erythrozyten zu klassifizieren. Die Ausgabe dieses künstlichen neuronalen Netzes wird durch manuelle Klassifizierung der Zellformen validiert und weist eine sehr hohe Übereinstimmung mit dieser manuellen Klassifikation auf. Im zweiten Teil der Arbeit untersuchen wir einen Effekt, der sich direkt aus der Deformierbarkeit der roten Blutzellen ergibt, das Lingering-Phänomen. Diese Lingering-Ereignisse können an Bifurkationsscheiteln zweier benachbarter Kapillaren auftreten und sind durch ein längeres Verweilen von Erythrozyten an einem Scheitelpunkt gekennzeichnet. In Simulationen hat sich gezeigt, dass diese Dynamik eine Anhäufung von nachfolgenden roten Blutzellen verursacht. Wir analysieren die Dynamik solcher Verweilereignisse in vivo, die wir folglich mit der Aufteilung von Erythrozyten an sich gabelnden Gefäßen in der Mikrovaskulatur in Verbindung bringen. Insbesondere verursacht das Verweilen von Erythrozyten einen erhöhten interzellulären Abstand zu weiter stromabwärts liegenden Erythrozyten und damit einen reduzierten Hämatokrit

Application of machine learning techniques to weather forecasting

Weather forecasting is, still today, a human based activity. Although computer simulations play a major role in modelling the state and evolution of the atmosphere, there is a lack of methodologies to automate the interpretation of the information generated by these models. This doctoral thesis explores the use of machine learning methodologies to solve specific problems in meteorology and particularly focuses on the exploration of methodologies to improve the accuracy of numerical weather prediction models using machine learning. The work presented in this manuscript contains two different approaches using machine learning. In the first part, classical methodologies, such as multivariate non-parametric regression and binary trees are explored to perform regression on meteorological data. In this first part, we particularly focus on forecasting wind, where the circular nature of this variable opens interesting challenges for classic machine learning algorithms and techniques. The second part of this thesis, explores the analysis of weather data as a generic structured prediction problem using deep neural networks. Neural networks, such as convolutional and recurrent networks provide a method for capturing the spatial and temporal structure inherent in weather prediction models. This part explores the potential of deep convolutional neural networks in solving difficult problems in meteorology, such as modelling precipitation from basic numerical model fields. The research performed during the completion of this thesis demonstrates that collaboration between the machine learning and meteorology research communities is mutually beneficial and leads to advances in both disciplines. Weather forecasting models and observational data represent unique examples of large (petabytes), structured and high-quality data sets, that the machine learning community demands for developing the next generation of scalable algorithms

Remote Sensing of the Aquatic Environments

The book highlights recent research efforts in the monitoring of aquatic districts with remote sensing observations and proximal sensing technology integrated with laboratory measurements. Optical satellite imagery gathered at spatial resolutions down to few meters has been used for quantitative estimations of harmful algal bloom extent and Chl-a mapping, as well as winds and currents from SAR acquisitions. The knowledge and understanding gained from this book can be used for the sustainable management of bodies of water across our planet