A Generalized EigenGame with Extensions to Multiview Representation Learning

Generalized Eigenvalue Problems (GEPs) encompass a range of interesting dimensionality reduction methods. Development of efficient stochastic approaches to these problems would allow them to scale to larger datasets. Canonical Correlation Analysis (CCA) is one example of a GEP for dimensionality reduction which has found extensive use in problems with two or more views of the data. Deep learning extensions of CCA require large mini-batch sizes, and therefore large memory consumption, in the stochastic setting to achieve good performance and this has limited its application in practice. Inspired by the Generalized Hebbian Algorithm, we develop an approach to solving stochastic GEPs in which all constraints are softly enforced by Lagrange multipliers. Then by considering the integral of this Lagrangian function, its pseudo-utility, and inspired by recent formulations of Principal Components Analysis and GEPs as games with differentiable utilities, we develop a game-theory inspired approach to solving GEPs. We show that our approaches share much of the theoretical grounding of the previous Hebbian and game theoretic approaches for the linear case but our method permits extension to general function approximators like neural networks for certain GEPs for dimensionality reduction including CCA which means our method can be used for deep multiview representation learning. We demonstrate the effectiveness of our method for solving GEPs in the stochastic setting using canonical multiview datasets and demonstrate state-of-the-art performance for optimizing Deep CCA

Human Motion Analysis for Efficient Action Recognition

Automatic understanding of human actions is at the core of several application domains, such as content-based indexing, human-computer interaction, surveillance, and sports video analysis. The recent advances in digital platforms and the exponential growth of video and image data have brought an urgent quest for intelligent frameworks to automatically analyze human motion and predict their corresponding action based on visual data and sensor signals. This thesis presents a collection of methods that targets human action recognition using different action modalities. The first method uses the appearance modality and classifies human actions based on heterogeneous global- and local-based features of scene and humanbody appearances. The second method harnesses 2D and 3D articulated human poses and analyizes the body motion using a discriminative combination of the parts’ velocities, locations, and correlations histograms for action recognition. The third method presents an optimal scheme for combining the probabilistic predictions from different action modalities by solving a constrained quadratic optimization problem. In addition to the action classification task, we present a study that compares the utility of different pose variants in motion analysis for human action recognition. In particular, we compare the recognition performance when 2D and 3D poses are used. Finally, we demonstrate the efficiency of our pose-based method for action recognition in spotting and segmenting motion gestures in real time from a continuous stream of an input video for the recognition of the Italian sign gesture language

Robust Rotation Synchronization via Low-rank and Sparse Matrix Decomposition

This paper deals with the rotation synchronization problem, which arises in global registration of 3D point-sets and in structure from motion. The problem is formulated in an unprecedented way as a "low-rank and sparse" matrix decomposition that handles both outliers and missing data. A minimization strategy, dubbed R-GoDec, is also proposed and evaluated experimentally against state-of-the-art algorithms on simulated and real data. The results show that R-GoDec is the fastest among the robust algorithms.Comment: The material contained in this paper is part of a manuscript submitted to CVI

Multiview pattern recognition methods for data visualization, embedding and clustering

Multiview data is defined as data for whose samples there exist several different data views, i.e. different data matrices obtained through different experiments, methods or situations. Multiview dimensionality reduction methods transform a high­dimensional, multiview dataset into a single, low-dimensional space or projection. Their goal is to provide a more manageable representation of the original data, either for data visualization or to simplify the following analysis stages. Multiview clustering methods receive a multiview dataset and propose a single clustering assignment of the data samples in the dataset, considering the information from all the input data views. The main hypothesis defended in this work is that using multiview data along with methods able to exploit their information richness produces better dimensionality reduction and clustering results than simply using single views or concatenating all views into a single matrix. Consequently, the objectives of this thesis are to develop and test multiview pattern recognition methods based on well known single-view dimensionality reduction and clustering methods. Three multiview pattern recognition methods are presented: multiview t-distributed stochastic neighbourhood embedding (MV-tSNE), multiview multimodal scaling (MV-MDS) and a novel formulation of multiview spectral clustering (MVSC-CEV). These methods can be applied both to dimensionality reduction tasks and to clustering tasks. The MV-tSNE method computes a matrix of probabilities based on distances between sam ples for each input view. Then it merges the different probability matrices using results from expert opinion pooling theory to get a common matrix of probabilities, which is then used as reference to build a low-dimensional projection of the data whose probabilities are similar. The MV-MDS method computes the common eigenvectors of all the normalized distance matrices in order to obtain a single low-dimensional space that embeds the essential information from all the input spaces, avoiding redundant information to be included. The MVSC-CEV method computes the symmetric Laplacian matrices of the similaritymatrices of all data views. Then it generates a single, low-dimensional representation of the input data by computing the common eigenvectors of the Laplacian matrices, obtaining a projection of the data that embeds the most relevan! information of the input data views, also avoiding the addition of redundant information. A thorough set of experiments has been designed and run in order to compare the proposed methods with their single view counterpart. Also, the proposed methods have been compared with all the available results of equivalent methods in the state of the art. Finally, a comparison between the three proposed methods is presented in order to provide guidelines on which method to use for a given task. MVSC-CEV consistently produces better clustering results than other multiview methods in the state of the art. MV-MDS produces overall better results than the reference methods in dimensionality reduction experiments. MV-tSNE does not excel on any of these tasks. As a consequence, for multiview clustering tasks it is recommended to use MVSC-CEV, and MV-MDS for multiview dimensionality reduction tasks. Although several multiview dimensionality reduction or clustering methods have been proposed in the state of the art, there is no software implementation available. In order to compensate for this fact and to provide the communitywith a potentially useful set of multiview pattern recognition methods, an R software package containg the proposed methods has been developed and released to the public.Los datos multivista se definen como aquellos datos para cuyas muestras existen varias vistas de datos distintas , es decir diferentes matrices de datos obtenidas mediante diferentes experimentos , métodos o situaciones. Los métodos multivista de reducción de la dimensionalidad transforman un conjunto de datos multivista y de alta dimensionalidad en un único espacio o proyección de baja dimensionalidad. Su objetivo es producir una representación más manejable de los datos originales, bien para su visualización o para simplificar las etapas de análisis subsiguientes. Los métodos de agrupamiento multivista reciben un conjunto de datos multivista y proponen una única asignación de grupos para sus muestras, considerando la información de todas las vistas de datos de entrada. La principal hipótesis defendida en este trabajo es que el uso de datos multivista junto con métodos capaces de aprovechar su riqueza informativa producen mejores resultados en reducción de la dimensionalidad y agrupamiento frente al uso de vistas únicas o la concatenación de varias vistas en una única matriz. Por lo tanto, los objetivos de esta tesis son desarrollar y probar métodos multivista de reconocimiento de patrones basados en métodos univista reconocidos. Se presentan tres métodos multivista de reconocimiento de patrones: proyección estocástica de vecinos multivista (MV-tSNE), escalado multidimensional multivista (MV-MDS) y una nueva formulación de agrupamiento espectral multivista (MVSC-CEV). Estos métodos pueden aplicarse tanto a tareas de reducción de la dimensionalidad como a de agrupamiento. MV-tSNE calcula una matriz de probabilidades basada en distancias entre muestras para cada vista de datos. A continuación combina las matrices de probabilidad usando resultados de la teoría de combinación de expertos para obtener una matriz común de probabilidades, que se usa como referencia para construir una proyección de baja dimensionalidad de los datos. MV-MDS calcula los vectores propios comunes de todas las matrices normalizadas de distancia para obtener un único espacio de baja dimensionalidad que integre la información esencial de todos los espacios de entrada, evitando información redundante. MVSC-CEVcalcula las matrices Laplacianas de las matrices de similitud de los datos. A continuación genera una única representación de baja dimensionalidad calculando los vectores propios comunes de las Laplacianas. Así obtiene una proyección de los datos que integra la información más relevante y evita añadir información redundante. Se ha diseñado y ejecutado una batería de experimentos completa para comparar los métodos propuestos con sus equivalentes univista. Además los métodos propuestos se han comparado con los resultados disponibles en la literatura. Finalmente, se presenta una comparación entre los tres métodos para proporcionar orientaciones sobre el método más adecuado para cada tarea. MVSC-CEV produce mejores agrupamientos que los métodos equivalentes en la literatura. MV-MDS produce en general mejores resultados que los métodos de referencia en experimentos de reducción de la dimensionalidad. MV-tSNE no destaca en ninguna de esas tareas . Consecuentemente , para agrupamiento multivista se recomienda usar MVSC-CEV, y para reducción de la dimensionalidad multivista MV-MDS. Aunque se han propuesto varios métodos multivista en la literatura, no existen programas disponibles públicamente. Para remediar este hecho y para dotar a la comunidad de un conjunto de métodos potencialmente útil, se ha desarrollado un paquete de programas en R y se ha puesto a disposición del público

Biview learning for human posture segmentation from 3D points cloud

Posture segmentation plays an essential role in human motion analysis. The state-of-the-art method extracts sufficiently high-dimensional features from 3D depth images for each 3D point and learns an efficient body part classifier. However, high-dimensional features are memory-consuming and difficult to handle on large-scale training dataset. In this paper, we propose an efficient two-stage dimension reduction scheme, termed biview learning, to encode two independent views which are depth-difference features (DDF) and relative position features (RPF). Biview learning explores the complementary property of DDF and RPF, and uses two stages to learn a compact yet comprehensive low-dimensional feature space for posture segmentation. In the first stage, discriminative locality alignment (DLA) is applied to the high-dimensional DDF to learn a discriminative low-dimensional representation. In the second stage, canonical correlation analysis (CCA) is used to explore the complementary property of RPF and the dimensionality reduced DDF. Finally, we train a support vector machine (SVM) over the output of CCA. We carefully validate the effectiveness of DLA and CCA utilized in the two-stage scheme on our 3D human points cloud dataset. Experimental results show that the proposed biview learning scheme significantly outperforms the state-of-the-art method for human posture segmentation. © 2014 Qiao et al