ADAPTIVE STEP-SIZES FOR REINFORCEMENT LEARNING

The central theme motivating this dissertation is the desire to develop reinforcement learning algorithms that “just work” regardless of the domain in which they are applied. The largest impediment to this goal is the sensitivity of reinforcement learning algorithms to the step-size parameter used to rescale incremental updates. Adaptive step-size algorithms attempt to reduce this sensitivity or eliminate the step-size parameter entirely by automatically adjusting the step size throughout the learning process. Such algorithms provide an alternative to the standard “guess-and-check” methods used to find parameters known as parameter tuning. However, the problems with parameter tuning are currently masked by the way experiments are conducted and presented. In this dissertation we seek algorithms that perform well over a broad subset of reinforcement learning problems with minimal parameter tuning. To accomplish this we begin by addressing the limitations of current empirical methods in reinforcement learning and propose improvements with benefits far outside the area of adaptive step-sizes. In order to study adaptive step-sizes in reinforcement learning we show that the general form of the adaptive step-size problem is a combination of two dissociable problems (adaptive scalar step-size and update whitening). We then derive new parameter-free adaptive scalar step-size algorithms for the reinforcement learning algorithm Sarsa(λ) and use our improved empirical methods to conduct a thorough experimental study of step-size algorithms in reinforcement learning. Our adaptive algorithms (VES and PARL2) both eliminate the need for a tunable step-size parameter and perform at least as well as Sarsa(λ) with an optimized step-size value. We conclude by developing natural temporal difference algorithms that provide an approximate solution to the update whitening problem and improve performance over their non-natural counterparts

Describing the Education Reform Landscape: A Typology of State Charter School Laws

Since 2014, 42 states have adopted charter school legislation. Research has been conducted on charter school effectiveness and legislative adoption. However, limitations in the research exist regarding school choice in that studies address inequalities and outcomes at the school level, with limited attention to the state-level policy environment. Additionally, research does not consider variations in state school choice policy nor does it link policy differences to equitable educational outcomes. This descriptive study described and categorized the variation of state charter school polices and explored differences in state level education finance, student demographics and academic outcomes, and school type characteristics. A cluster analysis yielded three clusters of states with charter school laws that were statistically and descriptively unique in terms of charter school autonomy, equity funding, and growth. ANOVA tests confirmed that the clusters were significantly different than one another. The three indices that were the basis of clustering have underlying composite variables that describe the nature of charter school laws in greater detail. Chi-square tests were conducted to determine whether or not the percentage of states, with each law characteristic specified in the composite variables that made up the index variables (autonomy, equity funding, and growth) differed significantly across clusters. Chi-square tests for all the composite variables reveal that the three state clusters differ significantly from one another. To further explore how the state clusters differed from one another in terms of factors examined in past research, the analysis compared cluster averages for variables measuring state level education finance, student demographics, education outcomes, and school types. ANOVAs were run for all of the clusters’ means for each characteristic variable. Only two of thirteen characteristic variables’ means were significantly different across clusters. The descriptive findings in this study can be used in concert with legislative adoption and charter school effectiveness research to reduce limitations in these research areas. Through this advance in charter school research, social workers will gain increased clarity to whether charter school reform is purportedly an equalizer of educational opportunity across class, race, ethnicity and/or gender

Science and civilization

An address delivered at the ninth commencement convocation of the Rice Institute, by Charles WIlliam Dabney, formerly President of the University of Cincinnati

Rlpy: A Value-Function-Based Reinforcement Learning Framework for Education and Research

RLPy is an object-oriented reinforcement learning software package with a focus on valuefunction-based methods using linear function approximation and discrete actions. The framework was designed for both educational and research purposes. It provides a rich library of fine-grained, easily exchangeable components for learning agents (e.g., policies or representations of value functions), facilitating recently increased specialization in reinforcement learning. RLPy is written in Python to allow fast prototyping, but is also suitable for large-scale experiments through its built-in support for optimized numerical libraries and parallelization. Code profiling, domain visualizations, and data analysis are integrated in a self-contained package available under the Modified BSD License at http://github.com/rlpy/rlpy. All of these properties allow users to compare various reinforcement learning algorithms with little effort

Tips on Establishing a Robotics Program in an Academic Setting

Over the past 5 years, robotic-assisted laparoscopic surgery has gone from being a novelty to an accepted approach for intra-abdominal and pelvic surgery. Driving this trend has been the large number of robotic-assisted laparoscopic prostatectomies performed throughout the U.S. Nearly a quarter of the prostatectomies done for prostate cancer in the U.S. in 2006 will use robotic assistance, yet reports fail to confirm cost effectiveness. The most important predictor of a successful program is a champion at the institution. Studies have demonstrated safety and immediate benefits with regard to reduced surgical morbidity such as pain, loss of work, quality of life, and blood loss for a variety of surgeries patients. Specific to prostatectomy for cancer, long-term data on biochemical (PSA) failures and cancer cures, as well as validated secondary outcomes for continence and potency, are still unavailable. Benefits accrue for the surgeon as well with improved ergonomics and potential extension of a surgical career. Yet, enthusiasm for robotics must be tempered by this lack of data and economic limitations. However, if a thoughtful and thorough process in initiating a robotic program is undertaken, the risks to the institution can be minimized. With proper training, the risk to the patient is reduced and with due diligence with regard to market and operative resources, the risk to the surgeon can be eliminated. This report reviews the steps to assess, plan, initiate, and maintain a robotics program at an academic institution with the hope that other programs can benefit from lessons acquired by early adopters of this expensive technology

Prophylactic methylprednisolone to reduce inflammation and improve outcomes from one lung ventilation in children: a randomized clinical trial.

BACKGROUND: One lung ventilation (OLV) results in inflammatory and mechanical injury, leading to intraoperative and postoperative complications in children. No interventions have been studied in children to minimize such injury. OBJECTIVE: We hypothesized that a single 2-mg·kg(-1) dose of methylprednisolone given 45-60 min prior to lung collapse would minimize injury from OLV and improve physiological stability. METHODS: Twenty-eight children scheduled to undergo OLV were randomly assigned to receive 2 mg·kg(-1) methylprednisolone (MP) or normal saline (placebo group) prior to OLV. Anesthetic management was standardized, and data were collected for physiological stability (bronchospasm, respiratory resistance, and compliance). Plasma was assayed for inflammatory markers related to lung injury at timed intervals related to administration of methylprednisolone. RESULTS: Three children in the placebo group experienced clinically significant intraoperative and postoperative respiratory complications. Respiratory resistance was lower (P = 0.04) in the methylprednisolone group. Pro-inflammatory cytokine IL-6 was lower (P = 0.01), and anti-inflammatory cytokine IL-10 was higher (P = 0.001) in the methylprednisolone group. Tryptase, measured before and after OLV, was lower (P = 0.03) in the methylprednisolone group while increased levels of tryptase were seen in placebo group after OLV (did not achieve significance). There were no side effects observed that could be attributed to methylprednisolone in this study. CONCLUSIONS: Methylprednisolone at 2 mg·kg(-1) given as a single dose prior to OLV provides physiological stability to children undergoing OLV. In addition, methylprednisolone results in lower pro-inflammatory markers and higher anti-inflammatory markers in the children\u27s plasma

Differential Pathogenesis of Lung Adenocarcinoma Subtypes Involving Sequence Mutations, Copy Number, Chromosomal Instability, and Methylation

Lung adenocarcinoma (LAD) has extreme genetic variation among patients, which is currently not well understood, limiting progress in therapy development and research. LAD intrinsic molecular subtypes are a validated stratification of naturally-occurring gene expression patterns and encompass different functional pathways and patient outcomes. Patients may have incurred different mutations and alterations that led to the different subtypes. We hypothesized that the LAD molecular subtypes co-occur with distinct mutations and alterations in patient tumors.The LAD molecular subtypes (Bronchioid, Magnoid, and Squamoid) were tested for association with gene mutations and DNA copy number alterations using statistical methods and published cohorts (n = 504). A novel validation (n = 116) cohort was assayed and interrogated to confirm subtype-alteration associations. Gene mutation rates (EGFR, KRAS, STK11, TP53), chromosomal instability, regional copy number, and genomewide DNA methylation were significantly different among tumors of the molecular subtypes. Secondary analyses compared subtypes by integrated alterations and patient outcomes. Tumors having integrated alterations in the same gene associated with the subtypes, e.g. mutation, deletion and underexpression of STK11 with Magnoid, and mutation, amplification, and overexpression of EGFR with Bronchioid. The subtypes also associated with tumors having concurrent mutant genes, such as KRAS-STK11 with Magnoid. Patient overall survival, cisplatin plus vinorelbine therapy response and predicted gefitinib sensitivity were significantly different among the subtypes.The lung adenocarcinoma intrinsic molecular subtypes co-occur with grossly distinct genomic alterations and with patient therapy response. These results advance the understanding of lung adenocarcinoma etiology and nominate patient subgroups for future evaluation of treatment response