Articulated human tracking and behavioural analysis in video sequences

Recently, there has been a dramatic growth of interest in the observation and tracking of human subjects through video sequences. Arguably, the principal impetus has come from the perceived demand for technological surveillance, however applications in entertainment, intelligent domiciles and medicine are also increasing. This thesis examines human articulated tracking and the classi cation of human movement, rst separately and then as a sequential process. First, this thesis considers the development and training of a 3D model of human body structure and dynamics. To process video sequences, an observation model is also designed with a multi-component likelihood based on edge, silhouette and colour. This is de ned on the articulated limbs, and visible from a single or multiple cameras, each of which may be calibrated from that sequence. Second, for behavioural analysis, we develop a methodology in which actions and activities are described by semantic labels generated from a Movement Cluster Model (MCM). Third, a Hierarchical Partitioned Particle Filter (HPPF) was developed for human tracking that allows multi-level parameter search consistent with the body structure. This tracker relies on the articulated motion prediction provided by the MCM at pose or limb level. Fourth, tracking and movement analysis are integrated to generate a probabilistic activity description with action labels. The implemented algorithms for tracking and behavioural analysis are tested extensively and independently against ground truth on human tracking and surveillance datasets. Dynamic models are shown to predict and generate synthetic motion, while MCM recovers both periodic and non-periodic activities, de ned either on the whole body or at the limb level. Tracking results are comparable with the state of the art, however the integrated behaviour analysis adds to the value of the approach.Overseas Research Students Awards Scheme (ORSAS

Object Localization and Recognition for Mobile Robots with Online Learning based on Mixture of Projected Gaussian

One of the primary capabilities required by autonomous robots is recognizing the surrounding environment with high responsiveness, often combined with object recognition and grasping tasks. Moreover robots acting in mutable scenarios are also required to be capable of learning new object models online. Along with peculiar requirements the robotics offers to the object recognition task some unique advantages, such the robot capability to move in the environment. Moreover, usually an autonomous robot can relax the recognition precision obtained at the beginning of its exploration and favour the speed at which this results are obtained. The aim of the work presented in this thesis is to explore a new object recognition method able to exploit this advantages in order to fulfil the features required by autonomous robotics. In order enhance pose estimation the proposed algorithm prioritize the keeping of the geometrical information from the objects shape and texture. Since the object models also need to be as much lightweight as possible this algorithm relies on local 6 DoF features extraction to describe the object appearance without load the final model of unnecessary information. Once the 6 DoF keypoints are obtained, the proposed method makes the use specifically designed probability distribution, namely the the Mixture of Projected Gaussian (MoPG) in order to learn their spatial distribution. A Bag of Words (BoW) technique has been introduced after the feature detection in order make feature descriptors more invariant to small appearance changes, due to light conditions or perspective distortions. The choice of using the MoPG distribution lies in one algebraic property of the Gaussian function, namely its closure over the convolution operator. In this thesis this property is exploited in order to obtain a closed form formula for calculating the cross-correlation of MoPG. The recognition algorithm makes use of the cross-correlation between MoPG in order to both identify and localize objects in the scene. The recognition and localization performances of the proposed technique was validated on two different publicly available datasets, namely the RGB-D Dataset and the BigBIRD Dataset. An analysis of both category and instance recognition results is presented and the emerged advantages or the issues of the proposed technique are discussed. The localization error (2 degrees) and the instance recognition rate (91%) resulted being aligned of the state of art thus justifying a further exploration of the proposed method. The topics presented in this thesis was further explored in some related works. In particular a collaboration with the Intelligent Systems Research Institute (Sungkyunkwan University, Republic of Corea) led an adapted version of the proposed method that has been successfully integrated in an autonomous domestic robot

Engineering limit cycle systems:adaptive frequency oscillators and applications to adaptive locomotion control of compliant robots

In this thesis, we present a dynamical systems approach to adaptive controllers for locomotion control. The approach is based on a rigorous mathematical framework using nonlinear dynamical systems and is inspired by theories of self-organization. Nonlinear dynamical systems such as coupled oscillators are an interesting approach for the on-line generation of trajectories for robots with many degrees of freedom (e.g. legged locomotion). However, designing a nonlinear dynamical system to satisfy a given specification and goal is not an easy task, and, hitherto no methodology exists to approach this problem in a unified way. Nature presents us with satisfactory solutions for the coordination of many degrees of freedom. One central feature observed in biological subjects is the ability of the neural systems to exploit natural dynamics of the body to achieve efficient locomotion. In order to be able to exploit the body properties, adaptive mechanisms must be at work. Recent work has pointed out the importance of the mechanical system for efficient locomotion. Even more interestingly, such well suited mechanical systems do not need complicated control. Yet, in robotics, in most approaches, adaptive mechanisms are either missing or they are not based on a rigorous framework, i.e. they are based on heuristics and ad-hoc approaches. Over the last three decades there has been enormous progress in describing movement coordination with the help of Synergetic approaches. This has led to the formulation of a theoretical framework: the theory of dynamic patterns. This framework is mathematically rigorous and at the same time fully operational. However, it does not provide any guidelines for synthetic approaches as needed for the engineering of robots with many degrees of freedom, nor does it directly help to explain adaptive systems. We will show how we can extend the theoretical framework to build adaptive systems. For this purpose, we propose the use of multi-scale dynamical systems. The basic idea behind multi-scale dynamical systems is that a given dynamical system gets extended by additional slow dynamics of its parameters, i.e. some of the parameters become state variables. The advantages of the framework of multi-scale dynamical systems for adaptive controllers are 1) fully dynamic description, 2) no separation of learning algorithm and learning substrate, 3) no separation of learning trials or time windows, 4) mathematically rigorous, 5) low dimensional systems. However, in order to fully exploit the framework important questions have to be solved. Most importantly, methodologies for designing the feedback loops have to be found and important theoretical questions about stability and convergence properties of the devised systems have to be answered. In order to tackle this challenge, we first introduce an engineering view on designing nonlinear dynamical systems and especially oscillators. We will highlight the important differences and freedom that this engineering view introduces as opposed to a modeling one. We then apply this approach by first proposing a very simple adaptive toy-system, consisting of a dynamical system coupled to a spring-mass system. Due to its spring-mass dynamics, this system contains clear natural dynamics in the form of resonant frequencies. We propose a prototype adaptive multi-scale system, the adaptive frequency oscillator, which is able to adapt its intrinsic frequency to the resonant frequency of the body dynamics. After a small sidetrack to show that we can use adaptive frequency oscillators also for other applications than for adaptive controllers, namely for frequency analysis, we then come back to further investigation of the adaptive controller. We apply the same controller concept to a simple spring-mass hopper system. The spring-mass system consists of a body with two legs attached by rotational joints. The legs contain spring-damper elements. Finally, we present results of the implementation of the controller on a real robot, the experimental robot PUPPY II. This robot is a under-actuated robot with spring dynamics in the knee joints. It will be shown, that due to the appropriate simplification and concentration on relevant features in the toy-system the controller concepts works without a fundamental change on all systems from the toy system up to the real robot

Exploitation des données cartographiques pour la perception de véhicules intelligents

This thesis is situated in the domains of robotics and data fusion, and concerns geographic information systems. We study the utility of adding digital maps, which model the urban environment in which the vehicle evolves, as a virtual sensor improving the perception results. Indeed, the maps contain a phenomenal quantity of information about the environment : its geometry, topology and additional contextual information. In this work, we extract road surface geometry and building models in order to deduce the context and the characteristics of each detected object. Our method is based on an extension of occupancy grids : the evidential perception grids. It permits to model explicitly the uncertainty related to the map and sensor data. By this means, the approach presents also the advantage of representing homogeneously the data originating from various sources : lidar, camera or maps. The maps are handled on equal terms with the physical sensors. This approach allows us to add geographic information without imputing unduly importance to it, which is essential in presence of errors. In our approach, the information fusion result, stored in a perception grid, is used to predict the stateof environment on the next instant. The fact of estimating the characteristics of dynamic elements does not satisfy the hypothesis of static world. Therefore, it is necessary to adjust the level of certainty attributed to these pieces of information. We do so by applying the temporal discounting. Due to the fact that existing methods are not well suited for this application, we propose a family of discoun toperators that take into account the type of handled information. The studied algorithms have been validated through tests on real data. We have thus developed the prototypes in Matlab and the C++ software based on Pacpus framework. Thanks to them, we present the results of experiments performed in real conditions.La plupart des logiciels contrôlant les véhicules intelligents traite de la compréhension de la scène. De nombreuses méthodes existent actuellement pour percevoir les obstacles de façon automatique. La majorité d’entre elles emploie ainsi les capteurs extéroceptifs comme des caméras ou des lidars. Cette thèse porte sur les domaines de la robotique et de la fusion d’information et s’intéresse aux systèmes d’information géographique. Nous étudions ainsi l’utilité d’ajouter des cartes numériques, qui cartographient le milieu urbain dans lequel évolue le véhicule, en tant que capteur virtuel améliorant les résultats de perception. Les cartes contiennent en effet une quantité phénoménale d’information sur l’environnement : sa géométrie, sa topologie ainsi que d’autres informations contextuelles. Dans nos travaux, nous avons extrait la géométrie des routes et des modèles de bâtiments afin de déduire le contexte et les caractéristiques de chaque objet détecté. Notre méthode se base sur une extension de grilles d’occupations : les grilles de perception crédibilistes. Elle permet de modéliser explicitement les incertitudes liées aux données de cartes et de capteurs. Elle présente également l’avantage de représenter de façon uniforme les données provenant de différentes sources : lidar, caméra ou cartes. Les cartes sont traitées de la même façon que les capteurs physiques. Cette démarche permet d’ajouter les informations géographiques sans pour autant leur donner trop d’importance, ce qui est essentiel en présence d’erreurs. Dans notre approche, le résultat de la fusion d’information contenu dans une grille de perception est utilisé pour prédire l’état de l’environnement à l’instant suivant. Le fait d’estimer les caractéristiques des éléments dynamiques ne satisfait donc plus l’hypothèse du monde statique. Par conséquent, il est nécessaire d’ajuster le niveau de certitude attribué à ces informations. Nous y parvenons en appliquant l’affaiblissement temporel. Étant donné que les méthodes existantes n’étaient pas adaptées à cette application, nous proposons une famille d’opérateurs d’affaiblissement prenant en compte le type d’information traitée. Les algorithmes étudiés ont été validés par des tests sur des données réelles. Nous avons donc développé des prototypes en Matlab et des logiciels en C++ basés sur la plate-forme Pacpus. Grâce à eux nous présentons les résultats des expériences effectués en conditions réelles

Increased Automobile Fuel Efficiency and Synthetic Fuels: Alternatives for Reducing Oil Imports

A study by the Office of Technology Assessment (OTA0 that "assesses and compares increased automobile fuel efficiency and synthetic fuels production with respect to their potential to reduce conventional oil consumption, and their costs and impacts" (Foreward)

The 2nd International Electronic Conference on Applied Sciences

This book is focused on the works presented at the 2nd International Electronic Conference on Applied Sciences, organized by Applied Sciences from 15 to 31 October 2021 on the MDPI Sciforum platform. Two decades have passed since the start of the 21st century. The development of sciences and technologies is growing ever faster today than in the previous century. The field of science is expanding, and the structure of science is becoming ever richer. Because of this expansion and fine structure growth, researchers may lose themselves in the deep forest of the ever-increasing frontiers and sub-fields being created. This international conference on the Applied Sciences was started to help scientists conduct their own research into the growth of these frontiers by breaking down barriers and connecting the many sub-fields to cut through this vast forest. These functions will allow researchers to see these frontiers and their surrounding (or quite distant) fields and sub-fields, and give them the opportunity to incubate and develop their knowledge even further with the aid of this multi-dimensional network

Increased confidence in concept design through trade space exploration and multiobjective optimization

Thesis (S.M.)--Massachusetts Institute of Technology, Dept. of Aeronautics and Astronautics, 2008.Includes bibliographical references (p. 134-143).The growing size, complexity and demands of engineering systems requires paying greater attention to the initial design of the system concept. To improve the process by which concept design is carried out, this thesis develops an Engineering Framework for Concept Development. The Engineering Framework is applicable to a diverse range of concept design problems. It helps guide the otherwise haphazard process of the early stages of design to provide confidence that the chosen concept is superior to a large set of alternatives. Accompanying the Engineering Framework is a collection of tools which aid the designer in analyzing different options. Two tools in particular are demonstrated for their mutually beneficial characteristics: 1) Object-Process Network is used to explore the full space of options, revealing the relationships among design decisions and system performance, and 2) a particle.swarm optimization algorithm is implemented to efficiently search through the design space. The use of such an optimization algorithm becomes especially advantageous when higher fidelity models are included in the analysis because it is able to quickly identify the most favorable families of designs. The complementary approaches of exploring the entire trade space and then efficiently searching for the best groups of designs are shown to provide valuable insights in concept design problems. Two case study examples are presented as applications of the Engineering Framework and design tools. The first is an air-launched sounding rocket propulsion system design. The second is the design of a responsive disaster monitoring system. In each case, the use of the Engineering Framework and concept design tools give the designer increased confidence that quality concept designs have been identified.by Ryan Glenn Odegard.S.M

Lokales Lernen für visuell kontrollierte Roboter

In this thesis a new supervised function approximation technique called Hierarchical Network of Locally Arranged Models is proposed to aid the development of learning-based visual robotic systems. In a coherent framework the new approach offers various means to create modular solutions to learning problems. It is possible to built up heterogeneous hierarchies so that different subnetworks can rely on different information sources. Modularity is realized by an automatic division of the input space of the target function into local regions where non-redundant models perform the demanded mapping into the output space. The goal is to replace one complex global model by a set of simple local ones. E.g. non-linear functions should be approximated with a number of simple linear models. The advantage of locality is the reduction of complexity: simple local models can more robustly be established and more easily be analyzed. Global validity is ensured by local specialization. The presented approach relies essentially on two new contributions: means to define the so-called domains of the local models (i.e. the region of their validity) and algorithms to split up the input space in order to achieve good approximation quality. The suggested models for the domains have different flexibility so that the local regions can have various shapes. Two learning algorithms are developed of which the offine version works on a fixed training set that is acquired before the application of the network, while the online version is useful if the network should be continually refined during operation. Both algorithms follow the strategy to place more local models at these regions of the input space where good approximation of the target function is harder to achieve. Furthermore, mechanisms are proposed that unify domains in order to simplify created networks, that define the degree of cooperation and competition between the different local models and that automatically detect data outliers to secure the application of a network. The value of the new approach is validated with public benchmark tests where several competitors are outperformed. The second major topic of this thesis is the application of the new machine learning technique in an adaptive robot vision system. The task is solved to train an arm robot to play a shape sorter puzzle where blocks have to be inserted into holes. To do so, different software modules are developed that realize interleaving perception-action cycles that drive the robot w.r.t. visual feedback. A visual servoing algorithm is presented that offers a simple principle to learn robot movements. It is based on the acquisition of training samples which represent observations of correct robot moves. The new approach to machine learning - specifically its features that are uncommon for supervised learning techniques - proves useful to realize this robot vision system. The possibility to combine different information sources in a hierarchy of local models helps to introduce application specific knowledge into the trained models. The outlier detection mechanism triggers error feedback within the system. The online learning algorithm makes the robot system robust against changes of its environment

Directional Estimation for Robotic Beating Heart Surgery

In robotic beating heart surgery, a remote-controlled robot can be used to carry out the operation while automatically canceling out the heart motion. The surgeon controlling the robot is shown a stabilized view of the heart. First, we consider the use of directional statistics for estimation of the phase of the heartbeat. Second, we deal with reconstruction of a moving and deformable surface. Third, we address the question of obtaining a stabilized image of the heart

Viewpoint-Free Photography for Virtual Reality

Viewpoint-free photography, i.e., interactively controlling the viewpoint of a photograph after capture, is a standing challenge. In this thesis, we investigate algorithms to enable viewpoint-free photography for virtual reality (VR) from casual capture, i.e., from footage easily captured with consumer cameras. We build on an extensive body of work in image-based rendering (IBR). Given images of an object or scene, IBR methods aim to predict the appearance of an image taken from a novel perspective. Most IBR methods focus on full or near-interpolation, where the output viewpoints either lie directly between captured images, or nearby. These methods are not suitable for VR, where the user has significant range of motion and can look in all directions. Thus, it is essential to create viewpoint-free photos with a wide field-of-view and sufficient positional freedom to cover the range of motion a user might experience in VR. We focus on two VR experiences: 1) Seated VR experiences, where the user can lean in different directions. This simplifies the problem, as the scene is only observed from a small range of viewpoints. Thus, we focus on easy capture, showing how to turn panorama-style capture into 3D photos, a simple representation for viewpoint-free photos, and also how to speed up processing so users can see the final result on-site. 2) Room-scale VR experiences, where the user can explore vastly different perspectives. This is challenging: More input footage is needed, maintaining real-time display rates becomes difficult, view-dependent appearance and object backsides need to be modelled, all while preventing noticeable mistakes. We address these challenges by: (1) creating refined geometry for each input photograph, (2) using a fast tiled rendering algorithm to achieve real-time display rates, and (3) using a convolutional neural network to hide visual mistakes during compositing. Overall, we provide evidence that viewpoint-free photography is feasible from casual capture. We thoroughly compare with the state-of-the-art, showing that our methods achieve both a numerical improvement and a clear increase in visual quality for both seated and room-scale VR experiences