Identifiability in inverse reinforcement learning

Inverse reinforcement learning attempts to reconstruct the reward function in a Markov decision problem, using observations of agent actions. As already observed in Russell [1998] the problem is ill-posed, and the reward function is not identifiable, even under the presence of perfect information about optimal behavior. We provide a resolution to this non-identifiability for problems with entropy regularization. For a given environment, we fully characterize the reward functions leading to a given policy and demonstrate that, given demonstrations of actions for the same reward under two distinct discount factors, or under sufficiently different environments, the unobserved reward can be recovered up to a constant. We also give general necessary and sufficient conditions for reconstruction of time-homogeneous rewards on finite horizons, and for action-independent rewards, generalizing recent results of Kim et al. [2021] and Fu et al. [2018]

Urania: Visualizing Data Analysis Pipelines for Natural Language-Based Data Exploration

Exploratory Data Analysis (EDA) is an essential yet tedious process for examining a new dataset. To facilitate it, natural language interfaces (NLIs) can help people intuitively explore the dataset via data-oriented questions. However, existing NLIs primarily focus on providing accurate answers to questions, with few offering explanations or presentations of the data analysis pipeline used to uncover the answer. Such presentations are crucial for EDA as they enhance the interpretability and reliability of the answer, while also helping users understand the analysis process and derive insights. To fill this gap, we introduce Urania, a natural language interactive system that is able to visualize the data analysis pipelines used to resolve input questions. It integrates a natural language interface that allows users to explore data via questions, and a novel data-aware question decomposition algorithm that resolves each input question into a data analysis pipeline. This pipeline is visualized in the form of a datamation, with animated presentations of analysis operations and their corresponding data changes. Through two quantitative experiments and expert interviews, we demonstrated that our data-aware question decomposition algorithm outperforms the state-of-the-art technique in terms of execution accuracy, and that Urania can help people explore datasets better. In the end, we discuss the observations from the studies and the potential future works

Combined early palliative care for non-small-cell lung cancer patients: a randomized controlled trial in Chongqing, China

PurposeMore effective approaches are needed to improve the prognosis of non-small-cell lung cancer (NSCLC) patients. Thus, we used the E-warm model to assess how early integration of interdisciplinary palliative care was related to the quality of life (QoL), psychological functioning, pain management, and nutrition factors of NSCLC patients.MethodsThis randomized controlled trial enrolled 280 newly diagnosed NSCLC patients, which were randomly divided (1:1) into combined early palliative care (CEPC) and standard oncological care (SC) groups. At baseline and after 24 weeks, the Functional Assessment of Cancer Therapy-Lung (FACT-L) scale, Hospital Anxiety and Depression Scale (HADS), and the Patient Health Questionnaire-9 (PHQ-9) were used to assess QoL and psychological function, respectively. The Numerical Rating Scale (NRS) and Patient-Generated Subjective Global Assessment (PG-SGA) were used to assess cancer patients’ pain and nutrition levels. The primary outcome was overall survival (OS). Secondary outcomes comprised changes in the QoL, psychological functioning, pain, and nutrition state. The intention-to-treat method was applied for analysis. This study was registered at www.chictr.org.cn (ChiCTR2200062617).ResultsOf the 140 patients enrolled in the CEPC and SC groups, 102 and 82 completed the research. The CEPC group presented higher QoL than the SC group (p < 0.05). Additionally, fewer patients presented depressive symptoms in the CEPC group than in the SC group (p < 0.05), as well as better nutritional status (p = 0.007) and pain management (p = 0.003). Compared to the SC group, CEPC patients had significantly longer OS (20.4 vs. 24.6 months, p = 0.042; HR: 0.19; 95% CI: 0.04-0.85, p = 0.029).ConclusionWith combined early palliative care, NSCLC patients lived longer, had better QoL, were psychologically stable, were in less pain, and were more nutritionally satisfied

arrayMap: A Reference Resource for Genomic Copy Number Imbalances in Human Malignancies

Background: The delineation of genomic copy number abnormalities (CNAs) from cancer samples has been instrumental for identification of tumor suppressor genes and oncogenes and proven useful for clinical marker detection. An increasing number of projects have mapped CNAs using high-resolution microarray based techniques. So far, no single resource does provide a global collection of readily accessible oncoge- nomic array data. Methodology/Principal Findings: We here present arrayMap, a curated reference database and bioinformatics resource targeting copy number profiling data in human cancer. The arrayMap database provides a platform for meta-analysis and systems level data integration of high-resolution oncogenomic CNA data. To date, the resource incorporates more than 40,000 arrays in 224 cancer types extracted from several resources, including the NCBI's Gene Expression Omnibus (GEO), EBIs ArrayExpress (AE), The Cancer Genome Atlas (TCGA), publication supplements and direct submissions. For the majority of the included datasets, probe level and integrated visualization facilitate gene level and genome wide data re- view. Results from multi-case selections can be connected to downstream data analysis and visualization tools. Conclusions/Significance: To our knowledge, currently no data source provides an extensive collection of high resolution oncogenomic CNA data which readily could be used for genomic feature mining, across a representative range of cancer entities. arrayMap represents our effort for providing a long term platform for oncogenomic CNA data independent of specific platform considerations or specific project dependence. The online database can be accessed at http://www.arraymap.org.Comment: 17 pages, 5 inline figures, 3 tables, supplementary figures/tables split into 4 PDF files; manuscript submitted to PLoS ON

Connecting Mean-field Games and Generative Adversarial Networks

The theory of mean-field games (MFGs) belongs to a branch of game theory that studies a large population of (weakly) interacting players. It serves as an analytically feasible framework to approximate stochastic differential games when the number of players is large and detailed characterization of the interactions is computationally expensive. As an effective modeling tool, the theory of MFGs attracts the attention of a variety of application fields in economics, finance and engineering. On the computation front, the development of machine learning provides abundant computational methods of solving for MFGs, which is remarkably meaningful in practice. At the same time, people may also wonder if the theory of MFGs, or stochastic analysis in general, could benefit the machine learning community.This thesis starts with two MFG models, with singular and impulse types of controls, respectively. Theses two control types allow certain degrees of discontinuity, making them better mathematical models compared with regular controls where the interventions must be absolute continuous. However, due to the theoretical challenges brought by the discontinuous nature of the controls, these two models are less explored in existing literature compared with MFGs with regular controls. Both models are motivated by real-world problems. Explicit solutions to the MFGs are presented and shown to approximate Nash equilibria of the corresponding N-player games with an error of the order O(1⁄√N). Further analysis of the solutions reveals the game effect from interacting with the mean-field.Obtaining analytical solutions of MFGs is difficult in general. The thesis then turns to the computation side of MFGs and establish the connection with generative adversarial networks, a celebrated deep learning tool that enjoys tremendous empirical success since its introduction to the machine learning community. It first shows a conceptual connection between GANs and MFGs: MFGs have the structure of GANs, and GANs are MFGs under the Pareto Optimality criterion. Interpreting MFGs as GANs, on one hand, enables a GANs-based algorithm (MFGANs) to solve MFGs: one neural network (NN) for the backward HJB equation and one NN for the forward FP equation, with the two NNs trained in an adversarial way. Viewing GANs as MFGs, on the other hand, reveals a new and probabilistic aspect of GANs. This new perspective, moreover, leads to an analytical connection between GANs and Optimal Transport (OT) problems, and sufficient conditions for the minimax games of GANs to be reformulated in the framework of OT. Numerical experiments demonstrate superior performance of this proposed algorithm, especially in higher dimensional case, when compared with existing NN approaches.Finally, the thesis explores the possibility of enriching the theoretical understanding of the training of GANs from the perspective of stochastic analysis. It establishes approximations, with precise error bound analysis, for the training of GANs under stochastic gradient algorithms (SGAs). The approximations are in the form of coupled stochastic differential equations (SDEs). The analysis of the SDEs and the associated invariant measures yields conditions for the stability and the convergence of GANs training. Further analysis of the invariant measure for the coupled SDEs gives rise to a fluctuation-dissipation relations (FDRs) for GANs, revealing the trade-off of the loss landscape between the generator and the discriminator and providing guidance for learning rate scheduling

Stationary Discounted and Ergodic Mean Field Games of Singular Control

Cao H, Dianetti J, Ferrari G. Stationary Discounted and Ergodic Mean Field Games of Singular Control. Center for Mathematical Economics Working Papers. Vol 650. Bielefeld: Center for Mathematical Economics; 2021.We study stationary mean field games with singular controls in which the representative player interacts with a long-time weighted average of the population through a discounted and an erodic performance criterion. This class of games finds natural applications in the context of optimal productivity expansion in dynamic oligopolies. We prove existence and uniqueness of the mean field equilibria for the discounted and the ergodic games by showing the validity of an Abelian limit. The latter allows also to approximate Nash equilibria of - so far unexplored - symmetric N-player ergodic singular control games through the mean field equilibrium of the discounted game. Numerical examples finally illustrate in a case study the dependency of the mean field equilibria with respect to the parameters of the games