Structure Discovery in Mixed Order Hyper Networks

Background  Mixed Order Hyper Networks (MOHNs) are a type of neural network in which the interactions between inputs are modelled explicitly by weights that can connect any number of neurons. Such networks have a human readability that networks with hidden units lack. They can be used for regression, classification or as content addressable memories and have been shown to be useful as fitness function models in constraint satisfaction tasks. They are fast to train and, when their structure is fixed, do not suffer from local minima in the cost function during training. However, their main drawback is that the correct structure (which neurons to connect with weights) must be discovered from data and an exhaustive search is not possible for networks of over around 30 inputs.  Results  This paper presents an algorithm designed to discover a set of weights that satisfy the joint constraints of low training error and a parsimonious model. The combined structure discovery and weight learning process was found to be faster, more accurate and have less variance than training an MLP.  Conclusions  There are a number of advantages to using higher order weights rather than hidden units in a neural network but discovering the correct structure for those weights can be challenging. With the method proposed in this paper, the use of high order networks becomes tractable

Motion-capture-based hand gesture recognition for computing and control

This dissertation focuses on the study and development of algorithms that enable the analysis and recognition of hand gestures in a motion capture environment. Central to this work is the study of unlabeled point sets in a more abstract sense. Evaluations of proposed methods focus on examining their generalization to users not encountered during system training. In an initial exploratory study, we compare various classification algorithms based upon multiple interpretations and feature transformations of point sets, including those based upon aggregate features (e.g. mean) and a pseudo-rasterization of the capture space. We find aggregate feature classifiers to be balanced across multiple users but relatively limited in maximum achievable accuracy. Certain classifiers based upon the pseudo-rasterization performed best among tested classification algorithms. We follow this study with targeted examinations of certain subproblems. For the first subproblem, we introduce the a fortiori expectation-maximization (AFEM) algorithm for computing the parameters of a distribution from which unlabeled, correlated point sets are presumed to be generated. Each unlabeled point is assumed to correspond to a target with independent probability of appearance but correlated positions. We propose replacing the expectation phase of the algorithm with a Kalman filter modified within a Bayesian framework to account for the unknown point labels which manifest as uncertain measurement matrices. We also propose a mechanism to reorder the measurements in order to improve parameter estimates. In addition, we use a state-of-the-art Markov chain Monte Carlo sampler to efficiently sample measurement matrices. In the process, we indirectly propose a constrained k-means clustering algorithm. Simulations verify the utility of AFEM against a traditional expectation-maximization algorithm in a variety of scenarios. In the second subproblem, we consider the application of positive definite kernels and the earth mover\u27s distance (END) to our work. Positive definite kernels are an important tool in machine learning that enable efficient solutions to otherwise difficult or intractable problems by implicitly linearizing the problem geometry. We develop a set-theoretic interpretation of ENID and propose earth mover\u27s intersection (EMI). a positive definite analog to ENID. We offer proof of EMD\u27s negative definiteness and provide necessary and sufficient conditions for ENID to be conditionally negative definite, including approximations that guarantee negative definiteness. In particular, we show that ENID is related to various min-like kernels. We also present a positive definite preserving transformation that can be applied to any kernel and can be used to derive positive definite EMD-based kernels, and we show that the Jaccard index is simply the result of this transformation applied to set intersection. Finally, we evaluate kernels based on EMI and the proposed transformation versus ENID in various computer vision tasks and show that END is generally inferior even with indefinite kernel techniques. Finally, we apply deep learning to our problem. We propose neural network architectures for hand posture and gesture recognition from unlabeled marker sets in a coordinate system local to the hand. As a means of ensuring data integrity, we also propose an extended Kalman filter for tracking the rigid pattern of markers on which the local coordinate system is based. We consider fixed- and variable-size architectures including convolutional and recurrent neural networks that accept unlabeled marker input. We also consider a data-driven approach to labeling markers with a neural network and a collection of Kalman filters. Experimental evaluations with posture and gesture datasets show promising results for the proposed architectures with unlabeled markers, which outperform the alternative data-driven labeling method

Knowledge Augmented Machine Learning with Applications in Autonomous Driving: A Survey

The existence of representative datasets is a prerequisite of many successful artificial intelligence and machine learning models. However, the subsequent application of these models often involves scenarios that are inadequately represented in the data used for training. The reasons for this are manifold and range from time and cost constraints to ethical considerations. As a consequence, the reliable use of these models, especially in safety-critical applications, is a huge challenge. Leveraging additional, already existing sources of knowledge is key to overcome the limitations of purely data-driven approaches, and eventually to increase the generalization capability of these models. Furthermore, predictions that conform with knowledge are crucial for making trustworthy and safe decisions even in underrepresented scenarios. This work provides an overview of existing techniques and methods in the literature that combine data-based models with existing knowledge. The identified approaches are structured according to the categories integration, extraction and conformity. Special attention is given to applications in the field of autonomous driving

Discriminative learning for structured outputs and environments

Machine learning methods have had considerable success across a wide range of applications. Much of this success is due to the flexibility of learning algorithms and their ability to tailor themselves to the requirements of the particular problem. In this thesis we examine methods that seek to exploit the underlying structure of a problem and make the best possible use of the available data. We explore the structural nature of two different problems, binary classification under the uncertainty of input relationships, and multi-label output learning of Markov networks with unknown graph structures. From the input perspective, we focus on binary classification and the problems associated with learning from limited amounts of data. In particular we pay attention to moment based methods and investigate how to deal with the uncertainty surrounding the estimate of moments using either small or noisy training samples. We present a worst-case analysis and show how the high probability bounds on the deviation of the true moments from their empirical counterparts can be used to generate a regularisation scheme that takes into consideration the relative amount of information that is available for each class. This results in a binary classification algorithm that directly minimises the worst case future misclassification rate, whilst taking into consideration the possible errors in the moment estimates. This algorithm was shown to outperform a number of traditional approaches across a range of benchmark datasets, doing particularly well when training was limited to small amounts of data. This supports the idea that we can leverage the class specific regularisation scheme and take advantage of the uncertainty of the datasets when creating a predictor. Further encouragement for this approach was provided during the high-noise experiments, predicting the directional movement of popular currency pairs, where moment based methods outperformed those using the peripheral point of the class-conditional distributions. From the output perspective, we focus on the problem of multi-label output learning over Markov networks and present a novel large margin learning method that leverages the correlation between output labels. Our approach is agnostic to the output graph structure and it simultaneously learns the intrinsic structure of the outputs, whilst finding a large margin separator. Based upon the observation that the score function over the complete output graph is given by the expectation of the score function over all spanning trees, we formulate the problem as an L1-norm multiple kernel learning problem where each spanning tree over the complete output graph gives rise to a particular instance of a kernel. We show that this approach is comparable to state-of-the-art approaches on a number of benchmark multi-label learning problems. Furthermore, we show how this method can be applied to the problem of predicting the joint movement of a group of stocks, where we not only infer the directional movement of individual stocks but also uncover insights on the input-dependent relationships between them