Efficient Human Activity Recognition in Large Image and Video Databases

Vision-based human action recognition has attracted considerable interest in recent research for its applications to video surveillance, content-based search, healthcare, and interactive games. Most existing research deals with building informative feature descriptors, designing efficient and robust algorithms, proposing versatile and challenging datasets, and fusing multiple modalities. Often, these approaches build on certain conventions such as the use of motion cues to determine video descriptors, application of off-the-shelf classifiers, and single-factor classification of videos. In this thesis, we deal with important but overlooked issues such as efficiency, simplicity, and scalability of human activity recognition in different application scenarios: controlled video environment (e.g.~indoor surveillance), unconstrained videos (e.g.~YouTube), depth or skeletal data (e.g.~captured by Kinect), and person images (e.g.~Flicker). In particular, we are interested in answering questions like (a) is it possible to efficiently recognize human actions in controlled videos without temporal cues? (b) given that the large-scale unconstrained video data are often of high dimension low sample size (HDLSS) nature, how to efficiently recognize human actions in such data? (c) considering the rich 3D motion information available from depth or motion capture sensors, is it possible to recognize both the actions and the actors using only the motion dynamics of underlying activities? and (d) can motion information from monocular videos be used for automatically determining saliency regions for recognizing actions in still images

LEAST SQUARES SUPPORT TENSOR MACHINE

Abstract Least squares support vector machine (LS-SVM), as a variant of the standard support vector machine (SVM) operates directly on patterns represented by vector and obtains an analytical solution directly from solving a set of linear equations instead of quadratic programming (QP). Tensor representation is useful to reduce the overfitting problem in vector-based learning, and tensor-based algorithm requires a smaller set of decision variables as compared to vector-based approaches. Above properties make the tensor learning specially suited for small-sample-size (S3) problems. In this paper, we generalize the vectorbased learning algorithm least squares support vector machine to the tensor-based method least squares support tensor machine (LS-STM), which accepts tensors as input. Similar to LS-SVM, the classifier is obtained also by solving a system of linear equations rather than a QP. LS-STM is based on the tensor space, with tensor representation, the number of parameters estimated by LS-STM is less than the number of parameters estimated by LS-SVM, and avoids discarding a great deal of useful structural information. Experimental results on some benchmark datasets indicate that the performance of LS-STM is competitive in classification performance compared to LS-SVM

Bridging Vision and Dynamic Legged Locomotion

Legged robots have demonstrated remarkable advances regarding robustness and versatility in the past decades. The questions that need to be addressed in this field are increasingly focusing on reasoning about the environment and autonomy rather than locomotion only. To answer some of these questions visual information is essential. If a robot has information about the terrain it can plan and take preventive actions against potential risks. However, building a model of the terrain is often computationally costly, mainly because of the dense nature of visual data. On top of the mapping problem, robots need feasible body trajectories and contact sequences to traverse the terrain safely, which may also require heavy computations. This computational cost has limited the use of visual feedback to contexts that guarantee (quasi-) static stability, or resort to planning schemes where contact sequences and body trajectories are computed before starting to execute motions. In this thesis we propose a set of algorithms that reduces the gap between visual processing and dynamic locomotion. We use machine learning to speed up visual data processing and model predictive control to achieve locomotion robustness. In particular, we devise a novel foothold adaptation strategy that uses a map of the terrain built from on-board vision sensors. This map is sent to a foothold classifier based on a convolutional neural network that allows the robot to adjust the landing position of the feet in a fast and continuous fashion. We then use the convolutional neural network-based classifier to provide safe future contact sequences to a model predictive controller that optimizes target ground reaction forces in order to track a desired center of mass trajectory. We perform simulations and experiments on the hydraulic quadruped robots HyQ and HyQReal. For all experiments the contact sequences, the foothold adaptations, the control inputs and the map are computed and processed entirely on-board. The various tests show that the robot is able to leverage the visual terrain information to handle complex scenarios in a safe, robust and reliable manner

Bone remodeling simulations: challenges, problems and applications

La remodelación ósea es el mecanismo que regula la relación entre la morfología del hueso y sus cargas mecánicas externas. Se basa en el hecho de que el hueso se adapta a las condiciones mecánicas a las que está expuesto. Varios factores mecánicos y bioquímicos pueden regular la respuesta final de la remodelación ósea. De hecho, se considera que la remodelación ósea pretende alcanzar varios objetivos mecánicos: reparar el daño para reducir el riesgo de fractura y optimizar la rigidez y resistencia con el mínimo peso. Durante las últimas décadas, se han propuesto un gran número de leyes matemáticas implementadas numéricamente, pero la mayoría de ellas presentan diferentes problemas como la estabilidad, la convergencia o la dependencia de las condiciones iniciales. Por tanto, el objetivo principal de esta tesis es estudiar los modelos de remodelación ósea, mostrando sus retos, su problemática y su aplicación en el ámbito clínico. En primer lugar, se han revisado dos teorías clásicas de la remodelación ósea (conocidas como modelo de Stanford y modelo de Doblaré y García). En ambos casos, se propone un aspecto novedoso planteando que el estímulo homeostático de referencia no es constante, sino que depende localmente de la historia de carga que cada punto local está soportando. Como consecuencia directa de esta hipótesis, se demuestra que las inestabilidades numéricas que normalmente presentan estos algoritmos, pueden quedar resueltas, mejorando claramente los resultados finales. Esta metodología se aplicó a un modelo de elementos finitos 2D/3D mejorando la convergencia de la solución y asegurando su estabilidad numérica a largo plazo. Por otra parte, en un intento de dilucidar las características de adaptación mecánica del hueso en diferentes escalas, se plantea una relación a nivel órgano y a nivel de tejido que depende de un cambio en el estímulo homeostático de referencia acorde con la densidad aparente, mientras que se considera que la densidad de energía de deformación a nivel de tejido permanece invariante. Esta hipótesis mejora la unicidad de la solución y la hace independiente de las condiciones iniciales, ayudando también a su estabilidad numérica. Además, en esta tesis se aborda el modelado de paciente específico que es un tema que está adquiriendo cada vez más importancia. Una de las principales dificultades en la creación de modelos de paciente específico, es la determinación de las cargas que el hueso está realmente soportando. Los datos relativos a pacientes específicos, como la geometría ósea y la distribución de la densidad ósea, puede ser utilizados para determinar estas cargas. Por lo tanto, se ha estudiado la estimación de la cargas con tres diferentes técnicas matemáticas: regresión lineal, redes neuronales artificiales y máquinas de soporte vector. Estas técnicas se han aplicado a un ejemplo teórico para obtener las cargas a través de la densidad aparente que se predice con los modelos de remodelación ósea. Para concluir, la metodología desarrollada que combina modelos de remodelación ósea con redes neuronales se ha aplicado a la predicción de las cargas de cinco tibias de pacientes. Para ello, se han determinado la geometría y la distribución de la densidad a partir de un TAC y se han introducido los valores de densidad en el modelo previamente desarrollado, obteniendo así, las cargas específicas de las tibias de los pacientes. Con el fin de validar la capacidad de esta novedosa técnica, se han comparado las cargas obtenidas de la técnica propuesta con las cargas obtenidas en un análisis de marcha de dichos pacientes. Los errores obtenidos en las predicciones han sido menores de un 6 %. Por lo tanto, se puede concluir que la metodología aquí propuesta, permite determinar de forma aproximada las cargas que un hueso específico soporta.Bone remodeling is the mechanism that regulates the relationship between bone morphology and its external mechanical loads. It is based on the fact that bone adapts itself to the mechanical conditions to which it is exposed. Several mechanical and biochemical factors may regulate the final bone remodeling response. In fact, bone remodeling is hypothesized to achieve several mechanical objectives: repair damage to reduce the risk of fracture and optimize stiffness and strength with minimum weight. During recent decades, a great number of numerically implemented mathematical laws have been proposed, but most of them present different problems as stability, convergence or dependence of the initial conditions. Thus, the main scope of this Thesis is to study bone remodeling models, showing their challenges, their problematic and their applicability in the clinical setting. Firstly, we revisit two classical bone remodeling theories (Stanford model and Doblaré and García model). In both of them, the reference homeostatic stimulus is hypothesized that is not constant, but it is locally dependent on the loading history that each local point is effectively supporting. As a direct consequence of this assumption, we demonstrate that the numerical instabilities that all these algorithms normally present can be solved, clearly improving the final results. For this reason, we applied this methodology to 2D/3D finite element models. This contribution improves the convergence of the solution, leading to its numerical stability in the long-term. In an attempt to elucidate the features of bone adaptation at the di erent scales, we hypothesize that the relationship between the organ level and tissue level depends on the reference homeostatic stimulus changes according to the density and the tissue effective energy remains unchanged. This assumption improves the uniqueness of the solution, independently of the initial conditions selected and clearly helps in its numerical stability. In addition, patient-specific modeling is becoming increasingly important. One of the most challenging diffculties in creating patient-specific models is the determination of the specific load that the bone is really supporting. Real information related to specific patients, such as bone geometry and bone density distribution, can be used to determine patient loads. Therefore, we studied three different mathematical techniques: linear regression, artificial neural networks (ANN) and support vector machines (SVM). These techniques have been applied to a theoretical femur to obtain the load through the density that came from many bone remodeling simulations. Finally, the application of this novel methodology has been applied for the loading prediction of five real tibias. We are able to determine the subject-specific forces from CT data, from which we obtain bone geometry and density distribuviition of the five tibias. Then, the density values at certain bone regions have been introduced in the methodology developed that combines bone remodeling models and artificial neuronal networks (ANN) for obtaining the predicted subject-specific loads. Finally, in order to validate this novel technique for tibia loading predictions, we compare predicted loads with the loads obtained from the patientspecific musculoskeletal model. The errors between both loads were lower tan 6%. Therefore, the methodology proposed has been validate

Learning discriminative features for human motion understanding

Human motion understanding has attracted considerable interest in recent research for its applications to video surveillance, content-based search and healthcare. With different capturing methods, human motion can be recorded in various forms (e.g. skeletal data, video, image, etc.). Compared to the 2D video and image, skeletal data recorded by motion capture device contains full 3D movement information. To begin with, we first look into a gait motion analysis problem based on 3D skeletal data. We propose an automatic framework for identifying musculoskeletal and neurological disorders among older people based on 3D skeletal motion data. In this framework, a feature selection strategy and two new gait features are proposed to choose an optimal feature set from the input features to optimise classification accuracy. Due to self-occlusion caused by single shooting angle, 2D video and image are not able to record full 3D geometric information. Therefore, viewpoint variation dramatically affects the performance on lots of 2D based applications (e.g. arbitrary view action recognition and image-based 3D human shape reconstruction). Leveraging view-invariance from the 3D model is a popular idea to improve the performance on 2D computer vision problems. Therefore, in the second contribution, we adopt 3D models built with computer graphics technology to assist in solving the problem of arbitrary view action recognition. As a solution, a new transfer dictionary learning framework that utilises computer graphics technologies to synthesise realistic 2D and 3D training videos is proposed, which can project a real-world 2D video into a view-invariant sparse representation. In the third contribution, 3D models are utilised to build an end-to-end 3D human shape reconstruction system, which can recover the 3D human shape from a single image without any prior parametric model. In contrast to most existing methods that calculate 3D joint locations, the method proposed in this thesis can produce a richer and more useful point cloud based representation. Synthesised high-quality 2D images and dense 3D point clouds are used to train a CNN-based encoder and 3D regression module. It can be concluded that the methods introduced in this thesis try to explore human motion understanding from 3D to 2D. We investigate how to compensate for the lack of full geometric information in 2D based applications with view-invariance learnt from 3D models

Machine Learning Methods with Noisy, Incomplete or Small Datasets

In many machine learning applications, available datasets are sometimes incomplete, noisy or affected by artifacts. In supervised scenarios, it could happen that label information has low quality, which might include unbalanced training sets, noisy labels and other problems. Moreover, in practice, it is very common that available data samples are not enough to derive useful supervised or unsupervised classifiers. All these issues are commonly referred to as the low-quality data problem. This book collects novel contributions on machine learning methods for low-quality datasets, to contribute to the dissemination of new ideas to solve this challenging problem, and to provide clear examples of application in real scenarios

Online Optimization-based Gait Adaptation of Quadruped Robot Locomotion

Quadruped robots demonstrated extensive capabilities of traversing complex and unstructured environments. Optimization-based techniques gave a relevant impulse to the research on legged locomotion. Indeed, by designing the cost function and the constraints, we can guarantee the feasibility of a motion and impose high-level locomotion tasks, e.g., tracking of a reference velocity. This allows one to have a generic planning approach without the need to tailor a specific motion for each terrain, as in the heuristic case. In this context, Model Predictive Control (MPC) can compensate for model inaccuracies and external disturbances, thanks to the high-frequency replanning. The main objective of this dissertation is to develop a Nonlinear MPC (NMPC)-based locomotion framework for quadruped robots. The aim is to obtain an algorithm which can be extended to different robots and gaits; in addition, I sought to remove some assumptions generally done in the literature, e.g., heuristic reference generator and user-defined gait sequence. The starting point of my work is the definition of the Optimal Control Problem to generate feasible trajectories for the Center of Mass. It is descriptive enough to capture the linear and angular dynamics of the robot as a whole. A simplified model (Single Rigid Body Dynamics model) is used for the system dynamics, while a novel cost term maximizes leg mobility to improve robustness in the presence of nonflat terrain. In addition, to test the approach on the real robot, I dedicated particular effort to implementing both a heuristic reference generator and an interface for the controller, and integrating them into the controller framework developed previously by other team members. As a second contribution of my work, I extended the locomotion framework to deal with a trot gait. In particular, I generalized the reference generator to be based on optimization. Exploiting the Linear Inverted Pendulum model, this new module can deal with the underactuation of the trot when only two legs are in contact with the ground, endowing the NMPC with physically informed reference trajectories to be tracked. In addition, the reference velocities are used to correct the heuristic footholds, obtaining contact locations coherent with the motion of the base, even though they are not directly optimized. The model used by the NMPC receives as input the gait sequence, thus with the last part of my work I developed an online multi-contact planner and integrated it into the MPC framework. Using a machine learning approach, the planner computes the best feasible option, even in complex environments, in a few milliseconds, by ranking online a set of discrete options for footholds, i.e., which leg to move and where to step. To train the network, I designed a novel function, evaluated offline, which considers the value of the cost of the NMPC and robustness/stability metrics for each option. These methods have been validated with simulations and experiments over the three years. I tested the NMPC on the Hydraulically actuated Quadruped robot (HyQ) of the IIT’s Dynamic Legged Systems lab, performing omni-directional motions on flat terrain and stepping on a pallet (both static and relocated during the motion) with a crawl gait. The trajectory replanning is performed at high-frequency, and visual information of the terrain is included to traverse uneven terrain. A Unitree Aliengo quadruped robot is used to execute experiments with the trot gait. The optimization-based reference generator allows the robot to reach a fixed goal and recover from external pushes without modifying the structure of the NMPC. Finally, simulations with the Solo robot are performed to validate the neural network-based contact planning. The robot successfully traverses complex scenarios, e.g., stepping stones, with both walk and trot gaits, choosing the footholds online. The achieved results improved the robustness and the performance of the quadruped locomotion. High-frequency replanning, dealing with a fixed goal, recovering after a push, and the automatic selection of footholds could help the robots to accomplish important tasks for the humans, for example, providing support in a disaster response scenario or inspecting an unknown environment. In the future, the contact planning will be transferred to the real hardware. Possible developments foresee the optimization of the gait timings, i.e., stance and swing duration, and a framework which allows the automatic transition between gaits

Cancellous bone and theropod dinosaur locomotion. Part I—an examination of cancellous bone architecture in the hindlimb bones of theropods

This paper is the first of a three-part series that investigates the architecture of cancellous (‘spongy’) bone in the main hindlimb bones of theropod dinosaurs, and uses cancellous bone architectural patterns to infer locomotor biomechanics in extinct non-avian species. Cancellous bone is widely known to be highly sensitive to its mechanical environment, and has previously been used to infer locomotor biomechanics in extinct tetrapod vertebrates, especially primates. Despite great promise, cancellous bone architecture has remained little utilized for investigating locomotion in many other extinct vertebrate groups, such as dinosaurs. Documentation and quantification of architectural patterns across a whole bone, and across multiple bones, can provide much information on cancellous bone architectural patterns and variation across species. Additionally, this also lends itself to analysis of the musculoskeletal biomechanical factors involved in a direct, mechanistic fashion. On this premise, computed tomographic and image analysis techniques were used to describe and analyse the three-dimensional architecture of cancellous bone in the main hindlimb bones of theropod dinosaurs for the first time. A comprehensive survey across many extant and extinct species is produced, identifying several patterns of similarity and contrast between groups. For instance, more stemward non-avian theropods (e.g. ceratosaurs and tyrannosaurids) exhibit cancellous bone architectures more comparable to that present in humans, whereas species more closely related to birds (e.g. paravians) exhibit architectural patterns bearing greater similarity to those of extant birds. Many of the observed patterns may be linked to particular aspects of locomotor biomechanics, such as the degree of hip or knee flexion during stance and gait. A further important observation is the abundance of markedly oblique trabeculae in the diaphyses of the femur and tibia of birds, which in large species produces spiralling patterns along the endosteal surface. Not only do these observations provide new insight into theropod anatomy and behaviour, they also provide the foundation for mechanistic testing of locomotor hypotheses via musculoskeletal biomechanical modelling