POMDPs in Continuous Time and Discrete Spaces

Many processes, such as discrete event systems in engineering or population dynamics in biology, evolve in discrete space and continuous time. We consider the problem of optimal decision making in such discrete state and action space systems under partial observability. This places our work at the intersection of optimal filtering and optimal control. At the current state of research, a mathematical description for simultaneous decision making and filtering in continuous time with finite countable state and action spaces is still missing. In this paper, we give a mathematical description of a continuous-time POMDP. By leveraging optimal filtering theory we derive a HJB type equation that characterizes the optimal solution. Using techniques from deep learning we approximately solve the resulting partial integro-differential equation. We present (i) an approach solving the decision problem offline by learning an approximation of the value function and (ii) an online algorithm which provides a solution in belief space using deep reinforcement learning. We show the applicability on a set of toy examples which pave the way for future methods providing solutions for high dimensional problems.Comment: published at Conference on Neural Information Processing Systems (NeurIPS) 202

Materials Physics in the Quantum Realm

Überblick über das Projekt QuantiCoM und aktuelle Herausforderungen im Bereich der atomistischen Simulationen mit Quantencomputern

Probabilistic inverse optimal control with local linearization for non-linear partially observable systems

Inverse optimal control methods can be used to characterize behavior in sequential decision-making tasks. Most existing work, however, requires the control signals to be known, or is limited to fully-observable or linear systems. This paper introduces a probabilistic approach to inverse optimal control for stochastic non-linear systems with missing control signals and partial observability that unifies existing approaches. By using an explicit model of the noise characteristics of the sensory and control systems of the agent in conjunction with local linearization techniques, we derive an approximate likelihood for the model parameters, which can be computed within a single forward pass. We evaluate our proposed method on stochastic and partially observable version of classic control tasks, a navigation task, and a manual reaching task. The proposed method has broad applicability, ranging from imitation learning to sensorimotor neuroscience

Receding Horizon Curiosity

Sample-efficient exploration is crucial not only for discovering rewarding experiences but also for adapting to environment changes in a task-agnostic fashion. A principled treatment of the problem of optimal input synthesis for system identification is provided within the framework of sequential Bayesian experimental design. In this paper, we present an effective trajectory-optimization-based approximate solution of this otherwise intractable problem that models optimal exploration in an unknown Markov decision process (MDP). By interleaving episodic exploration with Bayesian nonlinear system identification, our algorithm takes advantage of the inductive bias to explore in a directed manner, without assuming prior knowledge of the MDP. Empirical evaluations indicate a clear advantage of the proposed algorithm in terms of the rate of convergence and the final model fidelity when compared to intrinsic-motivation-based algorithms employing exploration bonuses such as prediction error and information gain. Moreover, our method maintains a computational advantage over a recent model-based active exploration (MAX) algorithm, by focusing on the information gain along trajectories instead of seeking a global exploration policy. A reference implementation of our algorithm and the conducted experiments is publicly available

Introducing multiple-choice questions to promote learning for medical students: effect on exam performance in obstetrics and gynecology

Abstract Purpose Testing is required in medical education. The large number of exams that students face requires effective learning strategies. Various methods of improving knowledge retention and recall have been discussed, two of the most widely evaluated of which are test-enhanced learning and pause procedures. This study investigated the effect of voluntary multiple-choice questions on students’ performance. Methods In a prospective study from April 2013 to March 2015, 721 students were randomly assigned to receive supplementary online material only (control group) or additional multiple-choice questions (investigative group) accompanying lectures. Their performance in the final exam was evaluated. Results A total of 675 students were ultimately included, with 299 randomly assigned to the investigative group and 376 to the control group. Students in the investigative group scored significantly better in relation to grades and points (2.11 vs. 2.49; 33 vs 31.31; p < 0.05). The effect declined over time. Conclusion This is the first study of the use of voluntary multiple-choice questions to improve medical students’ performance. The results support test-enhanced learning and the feasibility of implementing multiple-choice questions in lectures

Gemalte Normalität - gemalte Normen - gemalte Kultur: Was sagen Zeichnungen von Familien über familienbezogene Leitbilder aus?

Die hier dokumentierte Studie ist aus einem zu Lehrzwecken durchgeführten empirisch-methodischen Versuch heraus entstanden. Sie geht der Frage nach, inwieweit von Zeichnungen einer Familie, um die Probanden gebeten werden, auf die dahinterliegenden persönlichen oder kulturellen Leitbilder von Familie geschlossen werden kann und - falls dies zutrifft - was sich aus den 36 analysierten Zeichnungen konkret hinsichtlich der Familienleitbilder in Deutschland schließen lässt. Die Studie wurde 2013 im Rahmen eines Seminars an der Universität Mainz durchgeführt. Sie belegt, dass Zeichnungen sehr wohl ein wertvolles empirisches Material und ein methodischer Zugang zur Analyse von Familienleitbildern sein können. Allerdings sollte eine solche Analyse sich möglichst nicht auf eine reine Bildinterpretation stützen, sondern diese Interpretation durch nachträgliche auf die Zeichnung bezogene qualitative Interviews stützen. Das Leitbild der Familie in Deutschland erscheint im Lichte der Analysen stark auf die bürgerliche Kernfamilie fokussiert, bestehend aus einem verheirateten Paar aus Frau und Mann sowie etwa zwei minderjährigen Kindern, darunter ein Junge und ein Mädchen. Auch Großeltern und Haustiere sind zuweilen Teil der Vorstellung von Familie. Familienmitglieder halten eng zusammen und sind einander in Liebe verbunden. Familie bietet einen Schutzraum des Privaten gegen die Sorgen und Nöte, die in Beruf, Schule oder andernorts erfahren werden, und ermöglicht den Familienmitgliedern so Unbeschwertheit und glückliche gemeinsame Stunden. Familienleben findet zuhause im Eigenheim statt oder in der Natur - in jedem Fall an friedlichen und schönen Orten. Eine Vielfalt von Familienformen findet sich in den Familienleitbildern der Deutschen nur vereinzelt wieder.The study documented here is the result of an empirical-methodical experiment carried out for teaching purposes. It explores the extent to which drawings of a family, asked of study participants, can be used to draw conclusions regarding the underlying personal or cultural conceptions of family and - if this is the case - what can be concluded from the 36 drawings analysed with regard to family conceptions in Germany. The study was conducted in 2013 as part of a seminar at the University of Mainz. It proves that drawings can indeed be a valuable empirical material and a methodical approach to the analysis of family conceptions. However, such an analysis should not be based on a pure image interpretation alone, but ideally be supported by subsequent qualitative interviews related to the drawing. In the light of the analyses, the conception of family in Germany appears to be strongly focused on the middle-class nuclear family, consisting of a married couple of woman and man and about two minor children, including a boy and a girl. Grandparents and pets are also sometimes part of the association. Family members stick closely together and are united in love. Family offers a shelter of privacy from the worries and hardships experienced at work, school or elsewhere, allowing family members to enjoy carefree and happy hours together. Family life takes place at home in one‘s own home or in nature - in any case in peaceful and beautiful places. A variety of family forms can only be found sporadically in the family conceptions of the Germans

The Ninth Data Release of the Sloan Digital Sky Survey: First Spectroscopic Data from the SDSS-III Baryon Oscillation Spectroscopic Survey

The Sloan Digital Sky Survey III (SDSS-III) presents the first spectroscopic data from the Baryon Oscillation Spectroscopic Survey (BOSS). This ninth data release (DR9) of the SDSS project includes 535,995 new galaxy spectra (median z=0.52), 102,100 new quasar spectra (median z=2.32), and 90,897 new stellar spectra, along with the data presented in previous data releases. These spectra were obtained with the new BOSS spectrograph and were taken between 2009 December and 2011 July. In addition, the stellar parameters pipeline, which determines radial velocities, surface temperatures, surface gravities, and metallicities of stars, has been updated and refined with improvements in temperature estimates for stars with T_eff<5000 K and in metallicity estimates for stars with [Fe/H]>-0.5. DR9 includes new stellar parameters for all stars presented in DR8, including stars from SDSS-I and II, as well as those observed as part of the SDSS-III Sloan Extension for Galactic Understanding and Exploration-2 (SEGUE-2). The astrometry error introduced in the DR8 imaging catalogs has been corrected in the DR9 data products. The next data release for SDSS-III will be in Summer 2013, which will present the first data from the Apache Point Observatory Galactic Evolution Experiment (APOGEE) along with another year of data from BOSS, followed by the final SDSS-III data release in December 2014.Comment: 9 figures; 2 tables. Submitted to ApJS. DR9 is available at http://www.sdss3.org/dr