A General Deep Learning Framework for Network Reconstruction and Dynamics Learning

Many complex processes can be viewed as dynamical systems on networks. However, in real cases, only the performances of the system are known, the network structure and the dynamical rules are not observed. Therefore, recovering latent network structure and dynamics from observed time series data are important tasks because it may help us to open the black box, and even to build up the model of a complex system automatically. Although this problem hosts a wealth of potential applications in biology, earth science, and epidemics etc., conventional methods have limitations. In this work, we introduce a new framework, Gumbel Graph Network (GGN), which is a model-free, data-driven deep learning framework to accomplish the reconstruction of both network connections and the dynamics on it. Our model consists of two jointly trained parts: a network generator that generating a discrete network with the Gumbel Softmax technique; and a dynamics learner that utilizing the generated network and one-step trajectory value to predict the states in future steps. We exhibit the universality of our framework on different kinds of time-series data: with the same structure, our model can be trained to accurately recover the network structure and predict future states on continuous, discrete, and binary dynamics, and outperforms competing network reconstruction methods.Comment: v

Deep Clustering with a Dynamic Autoencoder: From Reconstruction towards Centroids Construction

In unsupervised learning, there is no apparent straightforward cost function that can capture the significant factors of variations and similarities. Since natural systems have smooth dynamics, an opportunity is lost if an unsupervised objective function remains static during the training process. The absence of concrete supervision suggests that smooth dynamics should be integrated. Compared to classical static cost functions, dynamic objective functions allow to better make use of the gradual and uncertain knowledge acquired through pseudo-supervision. In this paper, we propose Dynamic Autoencoder (DynAE), a novel model for deep clustering that overcomes a clustering-reconstruction trade-off, by gradually and smoothly eliminating the reconstruction objective function in favor of a construction one. Experimental evaluations on benchmark datasets show that our approach achieves state-of-the-art results compared to the most relevant deep clustering methods

Virtual Battery Parameter Identification using Transfer Learning based Stacked Autoencoder

Recent studies have shown that the aggregated dynamic flexibility of an ensemble of thermostatic loads can be modeled in the form of a virtual battery. The existing methods for computing the virtual battery parameters require the knowledge of the first-principle models and parameter values of the loads in the ensemble. In real-world applications, however, it is likely that the only available information are end-use measurements such as power consumption, room temperature, device on/off status, etc., while very little about the individual load models and parameters are known. We propose a transfer learning based deep network framework for calculating virtual battery state of a given ensemble of flexible thermostatic loads, from the available end-use measurements. This proposed framework extracts first order virtual battery model parameters for the given ensemble. We illustrate the effectiveness of this novel framework on different ensembles of ACs and WHs.Comment: 8 pages, 6 figures, accepted to IEEE ICMLA 201

Deep learning for universal linear embeddings of nonlinear dynamics

Identifying coordinate transformations that make strongly nonlinear dynamics approximately linear is a central challenge in modern dynamical systems. These transformations have the potential to enable prediction, estimation, and control of nonlinear systems using standard linear theory. The Koopman operator has emerged as a leading data-driven embedding, as eigenfunctions of this operator provide intrinsic coordinates that globally linearize the dynamics. However, identifying and representing these eigenfunctions has proven to be mathematically and computationally challenging. This work leverages the power of deep learning to discover representations of Koopman eigenfunctions from trajectory data of dynamical systems. Our network is parsimonious and interpretable by construction, embedding the dynamics on a low-dimensional manifold that is of the intrinsic rank of the dynamics and parameterized by the Koopman eigenfunctions. In particular, we identify nonlinear coordinates on which the dynamics are globally linear using a modified auto-encoder. We also generalize Koopman representations to include a ubiquitous class of systems that exhibit continuous spectra, ranging from the simple pendulum to nonlinear optics and broadband turbulence. Our framework parametrizes the continuous frequency using an auxiliary network, enabling a compact and efficient embedding at the intrinsic rank, while connecting our models to half a century of asymptotics. In this way, we benefit from the power and generality of deep learning, while retaining the physical interpretability of Koopman embeddings.Comment: v2: added another example and further details (increase from 9 pages to 14 pages and increase from 4 figures to 16 figures

Unsupervised Feature Learning of Human Actions as Trajectories in Pose Embedding Manifold

An unsupervised human action modeling framework can provide useful pose-sequence representation, which can be utilized in a variety of pose analysis applications. In this work we propose a novel temporal pose-sequence modeling framework, which can embed the dynamics of 3D human-skeleton joints to a continuous latent space in an efficient manner. In contrast to end-to-end framework explored by previous works, we disentangle the task of individual pose representation learning from the task of learning actions as a trajectory in pose embedding space. In order to realize a continuous pose embedding manifold with improved reconstructions, we propose an unsupervised, manifold learning procedure named Encoder GAN, (or EnGAN). Further, we use the pose embeddings generated by EnGAN to model human actions using a bidirectional RNN auto-encoder architecture, PoseRNN. We introduce first-order gradient loss to explicitly enforce temporal regularity in the predicted motion sequence. A hierarchical feature fusion technique is also investigated for simultaneous modeling of local skeleton joints along with global pose variations. We demonstrate state-of-the-art transfer-ability of the learned representation against other supervisedly and unsupervisedly learned motion embeddings for the task of fine-grained action recognition on SBU interaction dataset. Further, we show the qualitative strengths of the proposed framework by visualizing skeleton pose reconstructions and interpolations in pose-embedding space, and low dimensional principal component projections of the reconstructed pose trajectories.Comment: Accepted at WACV 201

Controlled hierarchical filtering: Model of neocortical sensory processing

A model of sensory information processing is presented. The model assumes that learning of internal (hidden) generative models, which can predict the future and evaluate the precision of that prediction, is of central importance for information extraction. Furthermore, the model makes a bridge to goal-oriented systems and builds upon the structural similarity between the architecture of a robust controller and that of the hippocampal entorhinal loop. This generative control architecture is mapped to the neocortex and to the hippocampal entorhinal loop. Implicit memory phenomena; priming and prototype learning are emerging features of the model. Mathematical theorems ensure stability and attractive learning properties of the architecture. Connections to reinforcement learning are also established: both the control network, and the network with a hidden model converge to (near) optimal policy under suitable conditions. Falsifying predictions, including the role of the feedback connections between neocortical areas are made.Comment: Technical Report, 38 pages, 10 figure

Data recovery in computational fluid dynamics through deep image priors

One of the challenges encountered by computational simulations at exascale is the reliability of simulations in the face of hardware and software faults. These faults, expected to increase with the complexity of the computational systems, will lead to the loss of simulation data and simulation failure and are currently addressed through a checkpoint-restart paradigm. Focusing specifically on computational fluid dynamics simulations, this work proposes a method that uses a deep convolutional neural network to recover simulation data. This data recovery method (i) is agnostic to the flow configuration and geometry, (ii) does not require extensive training data, and (iii) is accurate for very different physical flows. Results indicate that the use of deep image priors for data recovery is more accurate than standard recovery techniques, such as the Gaussian process regression, also known as Kriging. Data recovery is performed for two canonical fluid flows: laminar flow around a cylinder and homogeneous isotropic turbulence. For data recovery of the laminar flow around a cylinder, results indicate similar performance between the proposed method and Gaussian process regression across a wide range of mask sizes. For homogeneous isotropic turbulence, data recovery through the deep convolutional neural network exhibits an error in relevant turbulent quantities approximately three times smaller than that for the Gaussian process regression,. Forward simulations using recovered data illustrate that the enstrophy decay is captured within 10% using the deep convolutional neural network approach. Although demonstrated specifically for data recovery of fluid flows, this technique can be used in a wide range of applications, including particle image velocimetry, visualization, and computational simulations of physical processes beyond the Navier-Stokes equations

Disentangling Motion, Foreground and Background Features in Videos

This paper introduces an unsupervised framework to extract semantically rich features for video representation. Inspired by how the human visual system groups objects based on motion cues, we propose a deep convolutional neural network that disentangles motion, foreground and background information. The proposed architecture consists of a 3D convolutional feature encoder for blocks of 16 frames, which is trained for reconstruction tasks over the first and last frames of the sequence. A preliminary supervised experiment was conducted to verify the feasibility of proposed method by training the model with a fraction of videos from the UCF-101 dataset taking as ground truth the bounding boxes around the activity regions. Qualitative results indicate that the network can successfully segment foreground and background in videos as well as update the foreground appearance based on disentangled motion features. The benefits of these learned features are shown in a discriminative classification task, where initializing the network with the proposed pretraining method outperforms both random initialization and autoencoder pretraining. Our model and source code are publicly available at https://imatge-upc.github.io/unsupervised-2017-cvprw/ .Comment: Poster presented at the CVPR 2017 Workshop Brave New Ideas for Motion Representations in Video

A learning framework for winner-take-all networks with stochastic synapses

Many recent generative models make use of neural networks to transform the probability distribution of a simple low-dimensional noise process into the complex distribution of the data. This raises the question of whether biological networks operate along similar principles to implement a probabilistic model of the environment through transformations of intrinsic noise processes. The intrinsic neural and synaptic noise processes in biological networks, however, are quite different from the noise processes used in current abstract generative networks. This, together with the discrete nature of spikes and local circuit interactions among the neurons, raises several difficulties when using recent generative modeling frameworks to train biologically motivated models. In this paper, we show that a biologically motivated model based on multi-layer winner-take-all (WTA) circuits and stochastic synapses admits an approximate analytical description. This allows us to use the proposed networks in a variational learning setting where stochastic backpropagation is used to optimize a lower bound on the data log likelihood, thereby learning a generative model of the data. We illustrate the generality of the proposed networks and learning technique by using them in a structured output prediction task, and in a semi-supervised learning task. Our results extend the domain of application of modern stochastic network architectures to networks where synaptic transmission failure is the principal noise mechanism

A Hybrid Data-driven Deep Learning Technique for Fluid-Structure Interaction

This paper is concerned with the development of a hybrid data-driven technique for unsteady fluid-structure interaction systems. The proposed data-driven technique combines the deep learning framework with a projection-based low-order modeling. While the deep learning provides low-dimensional approximations from datasets arising from black-box solvers, the projection-based model constructs the low-dimensional approximations by projecting the original high-dimensional model onto a low-dimensional subspace. Of particular interest of this paper is to predict the long time series of unsteady flow fields of a freely vibrating bluff-body subjected to wake-body synchronization. We consider convolutional neural networks (CNN) for the learning dynamics of wake-body interaction, which assemble layers of linear convolutions with nonlinear activations to automatically extract the low-dimensional flow features. Using the high-fidelity time series data from the stabilized finite element Navier-Stokes solver, we first project the dataset to a low-dimensional subspace by proper orthogonal decomposition (POD) technique. The time-dependent coefficients of the POD subspace are mapped to the flow field via a CNN with nonlinear rectification, and the CNN is iteratively trained using the stochastic gradient descent method to predict the POD time coefficient when a new flow field is fed to it. The time-averaged flow field, the POD basis vectors, and the trained CNN are used to predict the long time series of the flow fields and the flow predictions are quantitatively assessed with the full-order (high-dimensional) simulation data. The proposed POD-CNN model based on the data-driven approximation has a remarkable accuracy in the entire fluid domain including the highly nonlinear near wake region behind a freely vibrating bluff body.Comment: 10 pages, 6 figures, under consideration to be published in OMAE 201