Recovering Markov Models from Closed-Loop Data

Situations in which recommender systems are used to augument decision making are becoming prevalent in many application domains. Almost always, these prediction tools (recommenders) are created with a view to affecting behavioural change. Clearly, successful applications actuating behavioural change, affect the original model underpinning the predictor, leading to an inconsistency. This feedback loop is often not considered in standard so-called Big Data learning techniques which rely upon machine learning/statistical learning machinery. The objective of this paper is to develop tools that recover unbiased user models in the presence of recommenders. More specifically, we assume that we observe a time series which is a trajectory of a Markov chain ${R}$ modulated by another Markov chain ${S}$, i.e. the transition matrix of ${R}$ is unknown and depends on the current state of ${S}$. The transition matrix of the latter is also unknown. In other words, at each time instant, ${S}$ selects a transition matrix for ${R}$ within a given set which consists of known and unknown matrices. The state of ${S}$, in turn, depends on the current state of ${R}$ thus introducing a feedback loop. We propose an Expectation-Maximization (EM) type algorithm, which estimates the transition matrices of ${S}$ and ${R}$. Experimental results are given to demonstrate the efficacy of the approach

An Adaptive Markov Random Field for Structured Compressive Sensing

Exploiting intrinsic structures in sparse signals underpins the recent progress in compressive sensing (CS). The key for exploiting such structures is to achieve two desirable properties: generality (\ie, the ability to fit a wide range of signals with diverse structures) and adaptability (\ie, being adaptive to a specific signal). Most existing approaches, however, often only achieve one of these two properties. In this study, we propose a novel adaptive Markov random field sparsity prior for CS, which not only is able to capture a broad range of sparsity structures, but also can adapt to each sparse signal through refining the parameters of the sparsity prior with respect to the compressed measurements. To maximize the adaptability, we also propose a new sparse signal estimation where the sparse signals, support, noise and signal parameter estimation are unified into a variational optimization problem, which can be effectively solved with an alternative minimization scheme. Extensive experiments on three real-world datasets demonstrate the effectiveness of the proposed method in recovery accuracy, noise tolerance, and runtime.Comment: 13 pages, submitted to IEEE Transactions on Image Processin

Neural Network Based Nonlinear Weighted Finite Automata

Weighted finite automata (WFA) can expressively model functions defined over strings but are inherently linear models. Given the recent successes of nonlinear models in machine learning, it is natural to wonder whether ex-tending WFA to the nonlinear setting would be beneficial. In this paper, we propose a novel model of neural network based nonlinearWFA model (NL-WFA) along with a learning algorithm. Our learning algorithm is inspired by the spectral learning algorithm for WFAand relies on a nonlinear decomposition of the so-called Hankel matrix, by means of an auto-encoder network. The expressive power of NL-WFA and the proposed learning algorithm are assessed on both synthetic and real-world data, showing that NL-WFA can lead to smaller model sizes and infer complex grammatical structures from data.Comment: AISTATS 201

Stochastic Stability of Event-triggered Anytime Control

We investigate control of a non-linear process when communication and processing capabilities are limited. The sensor communicates with a controller node through an erasure channel which introduces i.i.d. packet dropouts. Processor availability for control is random and, at times, insufficient to calculate plant inputs. To make efficient use of communication and processing resources, the sensor only transmits when the plant state lies outside a bounded target set. Control calculations are triggered by the received data. If a plant state measurement is successfully received and while the processor is available for control, the algorithm recursively calculates a sequence of tentative plant inputs, which are stored in a buffer for potential future use. This safeguards for time-steps when the processor is unavailable for control. We derive sufficient conditions on system parameters for stochastic stability of the closed loop and illustrate performance gains through numerical studies.Comment: IEEE Transactions on Automatic Control, under revie

Recovering Robustness in Model-Free Reinforcement learning

Reinforcement learning (RL) is used to directly design a control policy using data collected from the system. This paper considers the robustness of controllers trained via model-free RL. The discussion focuses on the standard model-based linear quadratic Gaussian (LQG) problem as a special instance of RL. A simple example, originally formulated for LQG problems, is used to demonstrate that RL with partial observations can lead to poor robustness margins. It is proposed to recover robustness by introducing random perturbations at the system input during the RL training. The perturbation magnitude can be used to trade off performance for robustness. Two simple examples are presented to demonstrate the proposed method for enhancing robustness during RL training.Comment: Github Code Repository: https://github.com/kumaa001/RLRobustness (Note : The files have been named to match with the section names and number. The comments in the code explains the procedure step by step. The data from the .mat file could be pulled into the work-space to avoid the need for complete code execution

Spatial Disaggregation of Agricultural Production Data

In this paper we develop a dynamic data-consistent way for estimating agricultural land use choices at a disaggregate level (district-level), using more aggregate data (regional-level). The disaggregation procedure requires two steps. The first step consists in specifying and estimating a dynamic model of land use at the regional level. In the second step, we disaggregate outcomes of the aggregate model using maximum entropy (ME). The ME disaggregation procedure is applied to a sample of California data. The sample includes 6 districts located in Central Valley and 8 possible crops, namely: Alfalfa, Cotton, Field, Grain, Melons, Tomatoes, Vegetables and Subtropical. The disaggregation procedure enables the recovery of land use at the district-level with an out-sample prediction error of 16%. This result shows that the micro behavior, inferred from aggregate data with our disaggregation approach, seems to be consistent with observed behavior.Disaggregation, Bayesian method, Maximum entropy, Land use, Production Economics, C11, C44, Q12,

Sequential Source Coding for Stochastic Systems Subject to Finite Rate Constraints

In this paper, we revisit the sequential source coding framework to analyze fundamental performance limitations of discrete-time stochastic control systems subject to feedback data-rate constraints in finite-time horizon. The basis of our results is a new characterization of the lower bound on the minimum total-rate achieved by sequential codes subject to a total (across time) distortion constraint and a computational algorithm that allocates optimally the rate-distortion for any fixed finite-time horizon. This characterization facilitates the derivation of analytical, non-asymptotic, and finite-dimensional lower and upper bounds in two control-related scenarios. (a) A parallel time-varying Gauss-Markov process with identically distributed spatial components that is quantized and transmitted through a noiseless channel to a minimum mean-squared error (MMSE) decoder. (b) A time-varying quantized LQG closed-loop control system, with identically distributed spatial components and with a random data-rate allocation. Our non-asymptotic lower bound on the quantized LQG control problem, reveals the absolute minimum data-rates for (mean square) stability of our time-varying plant for any fixed finite time horizon. We supplement our framework with illustrative simulation experiments.Comment: 40 pages, 6 figure

Anchored Discrete Factor Analysis

We present a semi-supervised learning algorithm for learning discrete factor analysis models with arbitrary structure on the latent variables. Our algorithm assumes that every latent variable has an "anchor", an observed variable with only that latent variable as its parent. Given such anchors, we show that it is possible to consistently recover moments of the latent variables and use these moments to learn complete models. We also introduce a new technique for improving the robustness of method-of-moment algorithms by optimizing over the marginal polytope or its relaxations. We evaluate our algorithm using two real-world tasks, tag prediction on questions from the Stack Overflow website and medical diagnosis in an emergency department

A tutorial on recursive models for analyzing and predicting path choice behavior

The problem at the heart of this tutorial consists in modeling the path choice behavior of network users. This problem has been extensively studied in transportation science, where it is known as the route choice problem. In this literature, individuals' choice of paths are typically predicted using discrete choice models. This article is a tutorial on a specific category of discrete choice models called recursive, and it makes three main contributions: First, for the purpose of assisting future research on route choice, we provide a comprehensive background on the problem, linking it to different fields including inverse optimization and inverse reinforcement learning. Second, we formally introduce the problem and the recursive modeling idea along with an overview of existing models, their properties and applications. Third, we extensively analyze illustrative examples from different angles so that a novice reader can gain intuition on the problem and the advantages provided by recursive models in comparison to path-based ones

A Monte Carlo method for critical systems in infinite volume: the planar Ising model

In this paper we propose a Monte Carlo method for generating finite-domain marginals of critical distributions of statistical models in infinite volume. The algorithm corrects the problem of the long-range effects of boundaries associated to generating critical distributions on finite lattices. It uses the advantage of scale invariance combined with ideas of the renormalization group in order to construct a type of "holographic" boundary condition that encodes the presence of an infinite volume beyond it. We check the quality of the distribution obtained in the case of the planar Ising model by comparing various observables with their infinite-plane prediction. We accurately reproduce planar two-, three- and four-point functions of spin and energy operators. We also define a lattice stress-energy tensor, and numerically obtain the associated conformal Ward identities and the Ising central charge.Comment: 43 pages, 21 figure