Efficient Optimization of Loops and Limits with Randomized Telescoping Sums

We consider optimization problems in which the objective requires an inner loop with many steps or is the limit of a sequence of increasingly costly approximations. Meta-learning, training recurrent neural networks, and optimization of the solutions to differential equations are all examples of optimization problems with this character. In such problems, it can be expensive to compute the objective function value and its gradient, but truncating the loop or using less accurate approximations can induce biases that damage the overall solution. We propose randomized telescope (RT) gradient estimators, which represent the objective as the sum of a telescoping series and sample linear combinations of terms to provide cheap unbiased gradient estimates. We identify conditions under which RT estimators achieve optimization convergence rates independent of the length of the loop or the required accuracy of the approximation. We also derive a method for tuning RT estimators online to maximize a lower bound on the expected decrease in loss per unit of computation. We evaluate our adaptive RT estimators on a range of applications including meta-optimization of learning rates, variational inference of ODE parameters, and training an LSTM to model long sequences

Meta-PDE: Learning to Solve PDEs Quickly Without a Mesh

Partial differential equations (PDEs) are often computationally challenging to solve, and in many settings many related PDEs must be be solved either at every timestep or for a variety of candidate boundary conditions, parameters, or geometric domains. We present a meta-learning based method which learns to rapidly solve problems from a distribution of related PDEs. We use meta-learning (MAML and LEAP) to identify initializations for a neural network representation of the PDE solution such that a residual of the PDE can be quickly minimized on a novel task. We apply our meta-solving approach to a nonlinear Poisson's equation, 1D Burgers' equation, and hyperelasticity equations with varying parameters, geometries, and boundary conditions. The resulting Meta-PDE method finds qualitatively accurate solutions to most problems within a few gradient steps; for the nonlinear Poisson and hyper-elasticity equation this results in an intermediate accuracy approximation up to an order of magnitude faster than a baseline finite element analysis (FEA) solver with equivalent accuracy. In comparison to other learned solvers and surrogate models, this meta-learning approach can be trained without supervision from expensive ground-truth data, does not require a mesh, and can even be used when the geometry and topology varies between tasks

Blind Attacks on Machine Learners

Abstract The importance of studying the robustness of learners to malicious data is well established. While much work has been done establishing both robust estimators and effective data injection attacks when the attacker is omniscient, the ability of an attacker to provably harm learning while having access to little information is largely unstudied. We study the potential of a "blind attacker" to provably limit a learner's performance by data injection attack without observing the learner's training set or any parameter of the distribution from which it is drawn. We provide examples of simple yet effective attacks in two settings: firstly, where an "informed learner" knows the strategy chosen by the attacker, and secondly, where a "blind learner" knows only the proportion of malicious data and some family to which the malicious distribution chosen by the attacker belongs. For each attack, we analyze minimax rates of convergence and establish lower bounds on the learner's minimax risk, exhibiting limits on a learner's ability to learn under data injection attack even when the attacker is "blind"

word~river literary review (2010)

wordriver is a literary journal dedicated to the poetry, short fiction and creative nonfiction of adjuncts and part-time instructors teaching in our universities, colleges, and community colleges. Our premier issue was published in Spring 2009. We are always looking for work that demonstrates the creativity and craft of adjunct/part-time instructors in English and other disciplines. We reserve first publication rights and onetime anthology publication rights for all work published. We define adjunct instructors as anyone teaching part-time or full-time under a semester or yearly contract, nationwide and in any discipline. Graduate students teaching under part-time contracts during the summer or who have used up their teaching assistant time and are teaching with adjunct contracts for the remainder of their graduate program also are eligible.https://digitalscholarship.unlv.edu/word_river/1000/thumbnail.jp

word~river literary review (2011)

wordriver is a literary journal dedicated to the poetry, short fiction and creative nonfiction of adjuncts and part-time instructors teaching in our universities, colleges, and community colleges. Our premier issue was published in Spring 2009. We are always looking for work that demonstrates the creativity and craft of adjunct/part-time instructors in English and other disciplines. We reserve first publication rights and onetime anthology publication rights for all work published. We define adjunct instructors as anyone teaching part-time or full-time under a semester or yearly contract, nationwide and in any discipline. Graduate students teaching under part-time contracts during the summer or who have used up their teaching assistant time and are teaching with adjunct contracts for the remainder of their graduate program also are eligible.https://digitalscholarship.unlv.edu/word_river/1001/thumbnail.jp

Impact of opioid-free analgesia on pain severity and patient satisfaction after discharge from surgery: multispecialty, prospective cohort study in 25 countries

Background: Balancing opioid stewardship and the need for adequate analgesia following discharge after surgery is challenging. This study aimed to compare the outcomes for patients discharged with opioid versus opioid-free analgesia after common surgical procedures.Methods: This international, multicentre, prospective cohort study collected data from patients undergoing common acute and elective general surgical, urological, gynaecological, and orthopaedic procedures. The primary outcomes were patient-reported time in severe pain measured on a numerical analogue scale from 0 to 100% and patient-reported satisfaction with pain relief during the first week following discharge. Data were collected by in-hospital chart review and patient telephone interview 1 week after discharge.Results: The study recruited 4273 patients from 144 centres in 25 countries; 1311 patients (30.7%) were prescribed opioid analgesia at discharge. Patients reported being in severe pain for 10 (i.q.r. 1-30)% of the first week after discharge and rated satisfaction with analgesia as 90 (i.q.r. 80-100) of 100. After adjustment for confounders, opioid analgesia on discharge was independently associated with increased pain severity (risk ratio 1.52, 95% c.i. 1.31 to 1.76; P < 0.001) and re-presentation to healthcare providers owing to side-effects of medication (OR 2.38, 95% c.i. 1.36 to 4.17; P = 0.004), but not with satisfaction with analgesia (beta coefficient 0.92, 95% c.i. -1.52 to 3.36; P = 0.468) compared with opioid-free analgesia. Although opioid prescribing varied greatly between high-income and low- and middle-income countries, patient-reported outcomes did not.Conclusion: Opioid analgesia prescription on surgical discharge is associated with a higher risk of re-presentation owing to side-effects of medication and increased patient-reported pain, but not with changes in patient-reported satisfaction. Opioid-free discharge analgesia should be adopted routinely

Learned surrogates and stochastic gradients for accelerating numerical modeling, simulation, and design

Numerical methods, such as discretization-based methods for solving ODEs and PDEs, allow us to model and design complex devices, structures, and systems. However, this is often very costly in terms of both computation and the time of the expert who must specify the physical governing equations, the discretization, the solver, and all other aspects of the numerical model. This thesis presents work using deep learning and stochastic gradient estimation to speed up numerical modeling procedures. In the first chapter we provide a broad introduction to numerical modeling, discuss the motivation for using machine learning (and other approximate methods) to speed it up, and discuss a few of the many methods which have been developed to do so. In chapter 2 we present composable energy surrogates, in which neural surrogates are trained to model a potential energy in sub-components or sub-domains of a PDE, and then composed together to solve a larger system by minimizing the sum of potentials across components. This allows surrogate modeling without requiring the full system to be solved with an expensive ground-truth finite element solver to generate training data. Instead, training data are generated cheaply by performing finite element analysis with individual components. We show that these surrogates can accelerate simulation of parametric meta-materials and produce accurate macroscopic behavior when composed. In chapter 3 we discuss randomized telescoping gradient estimators, which provide unbiased gradient estimators for objectives which are the limit of a sequence of increasingly accurate, increasingly costly approximations -- as we often encounter in numerical modeling. These estimators represent the limit as a telescoping sum and sample linear combinations of terms to provide cheap unbiased estimates. We discuss conditions which permit finite variance and computation, optimality of certain estimators within this class, and application to problems in numerical modeling and machine learning. In chapter 4 we discuss meta-learned implicit PDE solvers, which allow a new API for surrogate modeling. These models condition on a functional representation of a PDE and its domain by directly taking as input the PDE constraint and a method which returns samples in the domain and on the boundary. This avoids having to fix a parametric representation for PDEs within the class for which we wish to fit a surrogate, and allows fitting surrogate models for PDEs with arbitrarily varying geometry and governing equations. In aggregate, the work in this thesis aims to take machine learning in numerical modeling beyond simple regression-based surrogate modeling, and instead tailor machine learning methods to exploit and dovetail with the computational and physical structure of numerical models. This allows methods which are more computationally and data-efficient, and which have less-restrictive APIs, which might better empower scientists and engineers