How to Enable Uncertainty Estimation in Proximal Policy Optimization

While deep reinforcement learning (RL) agents have showcased strong results across many domains, a major concern is their inherent opaqueness and the safety of such systems in real-world use cases. To overcome these issues, we need agents that can quantify their uncertainty and detect out-of-distribution (OOD) states. Existing uncertainty estimation techniques, like Monte-Carlo Dropout or Deep Ensembles, have not seen widespread adoption in on-policy deep RL. We posit that this is due to two reasons: concepts like uncertainty and OOD states are not well defined compared to supervised learning, especially for on-policy RL methods. Secondly, available implementations and comparative studies for uncertainty estimation methods in RL have been limited. To overcome the first gap, we propose definitions of uncertainty and OOD for Actor-Critic RL algorithms, namely, proximal policy optimization (PPO), and present possible applicable measures. In particular, we discuss the concepts of value and policy uncertainty. The second point is addressed by implementing different uncertainty estimation methods and comparing them across a number of environments. The OOD detection performance is evaluated via a custom evaluation benchmark of in-distribution (ID) and OOD states for various RL environments. We identify a trade-off between reward and OOD detection performance. To overcome this, we formulate a Pareto optimization problem in which we simultaneously optimize for reward and OOD detection performance. We show experimentally that the recently proposed method of Masksembles strikes a favourable balance among the survey methods, enabling high-quality uncertainty estimation and OOD detection while matching the performance of original RL agents

Patient safety in developing countries: retrospective estimation of scale and nature of harm to patients in hospital

OBJECTIVE: To assess the frequency and nature of adverse events to patients in selected hospitals in developing or transitional economies. DESIGN: Retrospective medical record review of hospital admissions during 2005 in eight countries. SETTING: Ministries of Health of Egypt, Jordan, Kenya, Morocco, Tunisia, Sudan, South Africa and Yemen; the World Health Organisation (WHO) Eastern Mediterranean and African Regions (EMRO and AFRO), and WHO Patient Safety. PARTICIPANTS: Convenience sample of 26 hospitals from which 15,548 patient records were randomly sampled. MAIN OUTCOME MEASURES: Two stage screening. Initial screening based on 18 explicit criteria. Records that screened positive were then reviewed by a senior physician for determination of adverse event, its preventability, and the resulting disability. RESULTS: Of the 15,548 records reviewed, 8.2% showed at least one adverse event, with a range of 2.5% to 18.4% per country. Of these events, 83% were judged to be preventable, while about 30% were associated with death of the patient. About 34% adverse events were from therapeutic errors in relatively non-complex clinical situations. Inadequate training and supervision of clinical staff or the failure to follow policies or protocols contributed to most events. CONCLUSIONS: Unsafe patient care represents a serious and considerable danger to patients in the hospitals that were studied, and hence should be a high priority public health problem. Many other developing and transitional economies will probably share similar rates of harm and similar contributory factors. The convenience sampling of hospitals might limit the interpretation of results, but the identified adverse event rates show an estimate that should stimulate and facilitate the urgent institution of appropriate remedial action and also to trigger more research. Prevention of these adverse events will be complex and involves improving basic clinical processes and does not simply depend on the provision of more resources

Predicting self‐declared movie watching behavior using Facebook data and information‐fusion sensitivity analysis

The main purpose of this paper is to evaluate the feasibility of predicting whether yes or no a Facebook user has self-reported to have watched a given movie genre. Therefore, we apply a data analytical framework that (1) builds and evaluates several predictive models explaining self-declared movie watching behavior, and (2) provides insight into the importance of the predictors and their relationship with self-reported movie watching behavior. For the first outcome, we benchmark several algorithms (logistic regression, random forest, adaptive boosting, rotation forest, and naive Bayes) and evaluate their performance using the area under the receiver operating characteristic curve. For the second outcome, we evaluate variable importance and build partial dependence plots using information-fusion sensitivity analysis for different movie genres. To gather the data, we developed a custom native Facebook app. We resampled our dataset to make it representative of the general Facebook population with respect to age and gender. The results indicate that adaptive boosting outperforms all other algorithms. Time- and frequency-based variables related to media (movies, videos, and music) consumption constitute the list of top variables. To the best of our knowledge, this study is the first to fit predictive models of self-reported movie watching behavior and provide insights into the relationships that govern these models. Our models can be used as a decision tool for movie producers to target potential movie-watchers and market their movies more efficiently

Why visualize? untangling a large network of arguments

Visualization has been deemed a useful technique by researchers and practitioners, alike, leaving a trail of arguments behind that reason why visualization works. In addition, examples of misleading usages of visualizations in information communication have occasionally been pointed out. Thus, to contribute to the fundamental understanding of our discipline, we require a comprehensive collection of arguments on "why visualize?" (or "why not?"), untangling the rationale behind positive and negative viewpoints. In this paper, we report a theoretical study to understand the underlying reasons of various arguments; their relationships (e.g., built-on, and conflict); and their respective dependencies on tasks, users, and data. We curated an argumentative network based on a collection of arguments from various fields, including information visualization, cognitive science, psychology, statistics, philosophy, and others. Our work proposes several categorizations for the arguments, and makes their relations explicit. We contribute the first comprehensive and systematic theoretical study of the arguments on visualization. Thereby, we provide a roadmap towards building a foundation for visualization theory and empirical research as well as for practical application in the critique and design of visualizations. In addition, we provide our argumentation network and argument collection online at https://whyvis.dbvis.de, supported by an interactive visualization

Speculative execution of similarity queries: Real-time parameter optimization through visual exploration

The parameters of complex analytical models often have an unpredictable influence on the models' results, rendering parameter tuning a non-intuitive task. By concurrently visualizing both the model and its results, visual analytics tackles this issue, supporting the user in understanding the connection between abstract model parameters and model results. We present a visual analytics system enabling result understanding and model refinement on a ranking-based similarity search algorithm. Our system (1) visualizes the results in a projection view, mapping their pair-wise similarity to screen distance, (2) indicates the influence of model parameters on the results, and (3) implements speculative execution to enable real-time iterative refinement on the time-intensive offline similarity search algorithm

Visualization and the digital humanities: Moving towards stronger collaborations

For the past two years, researchers from the visualization community and the digital humanities have come together at the IEEE VIS conference to discuss how both disciplines can work together to push research goals in their respective disciplines. In this paper, we present our experiences as a result of this collaboration

Visualization and the digital humanities: Moving towards stronger collaborations

For the past two years, researchers from the visualization community and the digital humanities have come together at the IEEE VIS conference to discuss how both disciplines can work together to push research goals in their respective disciplines. In this paper, we present our experiences as a result of this collaboration