Bayesian learning of noisy Markov decision processes

We consider the inverse reinforcement learning problem, that is, the problem of learning from, and then predicting or mimicking a controller based on state/action data. We propose a statistical model for such data, derived from the structure of a Markov decision process. Adopting a Bayesian approach to inference, we show how latent variables of the model can be estimated, and how predictions about actions can be made, in a unified framework. A new Markov chain Monte Carlo (MCMC) sampler is devised for simulation from the posterior distribution. This step includes a parameter expansion step, which is shown to be essential for good convergence properties of the MCMC sampler. As an illustration, the method is applied to learning a human controller

On particle Gibbs sampling

The particle Gibbs sampler is a Markov chain Monte Carlo (MCMC) algorithm to sample from the full posterior distribution of a state-space model. It does so by executing Gibbs sampling steps on an extended target distribution defined on the space of the auxiliary variables generated by an interacting particle system. This paper makes the following contributions to the theoretical study of this algorithm. Firstly, we present a coupling construction between two particle Gibbs updates from different starting points and we show that the coupling probability may be made arbitrarily close to one by increasing the number of particles. We obtain as a direct corollary that the particle Gibbs kernel is uniformly ergodic. Secondly, we show how the inclusion of an additional Gibbs sampling step that reselects the ancestors of the particle Gibbs' extended target distribution, which is a popular approach in practice to improve mixing, does indeed yield a theoretically more efficient algorithm as measured by the asymptotic variance. Thirdly, we extend particle Gibbs to work with lower variance resampling schemes. A detailed numerical study is provided to demonstrate the efficiency of particle Gibbs and the proposed variants.Comment: Published at http://dx.doi.org/10.3150/14-BEJ629 in the Bernoulli (http://isi.cbs.nl/bernoulli/) by the International Statistical Institute/Bernoulli Society (http://isi.cbs.nl/BS/bshome.htm

Sequential quasi-Monte Carlo: Introduction for Non-Experts, Dimension Reduction, Application to Partly Observed Diffusion Processes

SMC (Sequential Monte Carlo) is a class of Monte Carlo algorithms for filtering and related sequential problems. Gerber and Chopin (2015) introduced SQMC (Sequential quasi-Monte Carlo), a QMC version of SMC. This paper has two objectives: (a) to introduce Sequential Monte Carlo to the QMC community, whose members are usually less familiar with state-space models and particle filtering; (b) to extend SQMC to the filtering of continuous-time state-space models, where the latent process is a diffusion. A recurring point in the paper will be the notion of dimension reduction, that is how to implement SQMC in such a way that it provides good performance despite the high dimension of the problem.Comment: To be published in the proceedings of MCMQMC 201

Transcriptional Regulation of Dendritic Cell Diversity

Dendritic cells (DCs) are specialized antigen presenting cells that are exquisitely adapted to sense pathogens and induce the development of adaptive immune responses. They form a complex network of phenotypically and functionally distinct subsets. Within this network, individual DC subsets display highly specific roles in local immunosurveillance, migration, and antigen presentation. This division of labor amongst DCs offers great potential to tune the immune response by harnessing subset-specific attributes of DCs in the clinical setting. Until recently, our understanding of DC subsets has been limited and paralleled by poor clinical translation and efficacy. We have now begun to unravel how different DC subsets develop within a complex multilayered system. These findings open up exciting possibilities for targeted manipulation of DC subsets. Furthermore, ground-breaking developments overcoming a major translational obstacle – identification of similar DC populations in mouse and man – now sets the stage for significant advances in the field. Here we explore the determinants that underpin cellular and transcriptional heterogeneity within the DC network, how these influence DC distribution and localization at steady-state, and the capacity of DCs to present antigens via direct or cross-presentation during pathogen infection

Forest resampling for distributed sequential Monte Carlo

This paper brings explicit considerations of distributed computing architectures and data structures into the rigorous design of Sequential Monte Carlo (SMC) methods. A theoretical result established recently by the authors shows that adapting interaction between particles to suitably control the Effective Sample Size (ESS) is sufficient to guarantee stability of SMC algorithms. Our objective is to leverage this result and devise algorithms which are thus guaranteed to work well in a distributed setting. We make three main contributions to achieve this. Firstly, we study mathematical properties of the ESS as a function of matrices and graphs that parameterize the interaction amongst particles. Secondly, we show how these graphs can be induced by tree data structures which model the logical network topology of an abstract distributed computing environment. Thirdly, we present efficient distributed algorithms that achieve the desired ESS control, perform resampling and operate on forests associated with these trees

Harold Jeffreys's Theory of Probability Revisited

Published exactly seventy years ago, Jeffreys's Theory of Probability (1939) has had a unique impact on the Bayesian community and is now considered to be one of the main classics in Bayesian Statistics as well as the initiator of the objective Bayes school. In particular, its advances on the derivation of noninformative priors as well as on the scaling of Bayes factors have had a lasting impact on the field. However, the book reflects the characteristics of the time, especially in terms of mathematical rigor. In this paper we point out the fundamental aspects of this reference work, especially the thorough coverage of testing problems and the construction of both estimation and testing noninformative priors based on functional divergences. Our major aim here is to help modern readers in navigating in this difficult text and in concentrating on passages that are still relevant today.Comment: This paper commented in: [arXiv:1001.2967], [arXiv:1001.2968], [arXiv:1001.2970], [arXiv:1001.2975], [arXiv:1001.2985], [arXiv:1001.3073]. Rejoinder in [arXiv:0909.1008]. Published in at http://dx.doi.org/10.1214/09-STS284 the Statistical Science (http://www.imstat.org/sts/) by the Institute of Mathematical Statistics (http://www.imstat.org

Some discussions of D. Fearnhead and D. Prangle's Read Paper "Constructing summary statistics for approximate Bayesian computation: semi-automatic approximate Bayesian computation"

This report is a collection of comments on the Read Paper of Fearnhead and Prangle (2011), to appear in the Journal of the Royal Statistical Society Series B, along with a reply from the authors.Comment: 10 page

Chronic wounds consultation by telemedicine between a rehabilitation healthcare center and nursing home or home

Saint-Hélier Rehabilitation Center (pôle MPR Saint-Hélier), located in Rennes, has been selected for a regional telemedicine project in 2014 about chronic wounds.AimTo make care access easier for heavy disabilities patients in nursing homes or at home with chronic wounds.MethodThe members of TLM Pl@ies chronic team are specialist doctors and nurses for wounds. On request, the occupational therapist or dietician involve in the consultation (multidisciplinary approach). A secure videoconference (web) is used.ResultsSince July 2014, over 100 teleconsultations have been done. Targeted population is constituted by patients:– whose access to care is decreased due to moving difficulties;– of which the health care team is crossing difficulties in the care process (wound care but also disability, nutrition..).Seventy percent of requests come from the nursing home, 30% from homes (pressure ulcers stages 3 and 4, arterial ulcers, venous or mixed). Middle age: 78 years (20–101 years). Only 3 patients refused. Time to organize the teleconsultation is on average 13 days. Consultations last on average 25 minutes. In 30% of cases the teleconsultation is extended by a real live training time for the nurse at home guided by the TLM Pl@ies chroniques team. We evaluate professional satisfaction and technical satisfaction. Without teleconsultation, in 77% of cases transportation request for consultation would be made, in 5% hospitalization. In 18% no request would be done.Discussion/conclusionThese first results, encouraging, confirms the interest of specialized consultations in medico-social settings, and telemedicine can be an effective solution