Active learning with RESSPECT: Resource allocation for extragalactic astronomical transients

The authors would like to thank David Kirkby and Connor Sheere for insightful discussions. This work is part of the Recommendation System for Spectroscopic Followup (RESSPECT) project, governed by an inter-collaboration agreement signed between the Cosmostatistics Initiative (COIN) and the LSST Dark Energy Science Collaboration (DESC). This research is supported in part by the HPI Research Center in Machine Learning and Data Science at UC Irvine. EEOI and SS acknowledge financial support from CNRS 2017 MOMENTUM grant under the project Active Learning for Large Scale Sky Surveys. SGG and AKM acknowledge support by FCT under Project CRISP PTDC/FIS-AST-31546/2017. This work was partly supported by the Hewlett Packard Enterprise Data Science Institute (HPE DSI) at the University of Houston. DOJ is supported by a Gordon and Betty Moore Foundation postdoctoral fellowship at the University of California, Santa Cruz. Support for this work was provided by NASA through the NASA Hubble Fellowship grant HF2-51462.001 awarded by the Space Telescope Science Institute, which is operated by the Association of Universities for Research in Astronomy, Inc., for NASA, under contract NAS5-26555. BQ is supported by the International Gemini Observatory, a program of NSF's NOIRLab, which is managed by the Association of Universities for Research in Astronomy (AURA) under a cooperative agreement with the National Science Foundation, on behalf of the Gemini partnership of Argentina, Brazil, Canada, Chile, the Republic of Korea, and the United States of America. AIM acknowledges support from the Max Planck Society and the Alexander von Humboldt Foundation in the framework of the Max Planck-Humboldt Research Award endowed by the Federal Ministry of Education and Research. L.G. was funded by the European Union's Horizon 2020 research and innovation programme under the Marie Sklodowska-Curie grant agreement No. 839090. This work has been partially supported by the Spanish grant PGC2018-095317-B-C21 within the European Funds for Regional Development (FEDER).The recent increase in volume and complexity of available astronomical data has led to a wide use of supervised machine learning techniques. Active learning strategies have been proposed as an alternative to optimize the distribution of scarce labeling resources. However, due to the specific conditions in which labels can be acquired, fundamental assumptions, such as sample representativeness and labeling cost stability cannot be fulfilled. The Recommendation System for Spectroscopic followup (RESSPECT) project aims to enable the construction of optimized training samples for the Rubin Observatory Legacy Survey of Space and Time (LSST), taking into account a realistic description of the astronomical data environment. In this work, we test the robustness of active learning techniques in a realistic simulated astronomical data scenario. Our experiment takes into account the evolution of training and pool samples, different costs per object, and two different sources of budget. Results show that traditional active learning strategies significantly outperform random sampling. Nevertheless, more complex batch strategies are not able to significantly overcome simple uncertainty sampling techniques. Our findings illustrate three important points: 1) active learning strategies are a powerful tool to optimize the label-acquisition task in astronomy, 2) for upcoming large surveys like LSST, such techniques allow us to tailor the construction of the training sample for the first day of the survey, and 3) the peculiar data environment related to the detection of astronomical transients is a fertile ground that calls for the development of tailored machine learning algorithms.HPI Research Center in Machine Learning and Data Science at UC IrvineCNRS 2017 MOMENTUM grant under the project Active Learning for Large Scale Sky SurveysFCT under Project CRISP PTDC/FIS-AST-31546/2017Hewlett Packard Enterprise Data Science Institute (HPE DSI) at the University of HoustonGordon and Betty Moore Foundation postdoctoral fellowship at the University of California, Santa CruzSpace Telescope Science InstituteNational Aeronautics & Space Administration (NASA) HF2-51462.001 NAS5-26555International Gemini Observatory, a program of NSF's NOIRLabNational Science Foundation (NSF)Max Planck SocietyFoundation CELLEXAlexander von Humboldt FoundationEuropean Commission 839090Spanish grant within the European Funds for Regional Development (FEDER) PGC2018-095317-B-C2

Variational Methods for Optimal Experimental Design

In this work we study variational methods for Bayesian optimal experimental design (BOED). Experimentation is a cornerstone of science and is central to any major engineering effort. Often experiments require the use of substantial resources, from expensive equipment to limited researcher time; in addition, experiments can be dangerous or may be required to be completed in a given period of time. For these reasons, we prefer to conduct our experiments as efficiently as possible, acquiring as much information as we can given the resources available to us. Optimal experimental design (OED) is a sub-field of statistics focused on developing methods for accomplishing this goal. The OED problem is formulated by defining a utility function over designs and optimizing this function over the set of all feasible designs. We focus on the \emph{Expected Information Gain} (EIG), a widely used utility function with sound theoretical support. However, in practice the EIG is intractable to compute, and approximation strategies are required. We investigate the use of variational methods for this purpose and show substantial improvement over competing approximation techniques. A specific form of OED common in the field of machine learning (ML) is \emph{active learning} (AL). In the active learning framework, we would like to obtain a labeled dataset in order to train a supervised model. However, for all the reasons stated, labeling data points can be costly and again we should make efficient use of our labeling resources. We present a novel application of active learning to optimize spectroscopic follow up for large scale astronomical surveys. Finally, much of this work requires learning functions over sets which we know must satisfy certain properties (e.g., permutation invariance). We conclude the thesis by presenting a novel neural network architecture for predicting the astronomical class of individual objects in the same exposure using a neural architecture specifically designed to accommodate known inductive biases present in the data

Variational Methods for Optimal Experimental Design

In this work we study variational methods for Bayesian optimal experimental design (BOED). Experimentation is a cornerstone of science and is central to any major engineering effort. Often experiments require the use of substantial resources, from expensive equipment to limited researcher time; in addition, experiments can be dangerous or may be required to be completed in a given period of time. For these reasons, we prefer to conduct our experiments as efficiently as possible, acquiring as much information as we can given the resources available to us. Optimal experimental design (OED) is a sub-field of statistics focused on developing methods for accomplishing this goal. The OED problem is formulated by defining a utility function over designs and optimizing this function over the set of all feasible designs. We focus on the \emph{Expected Information Gain} (EIG), a widely used utility function with sound theoretical support. However, in practice the EIG is intractable to compute, and approximation strategies are required. We investigate the use of variational methods for this purpose and show substantial improvement over competing approximation techniques. A specific form of OED common in the field of machine learning (ML) is \emph{active learning} (AL). In the active learning framework, we would like to obtain a labeled dataset in order to train a supervised model. However, for all the reasons stated, labeling data points can be costly and again we should make efficient use of our labeling resources. We present a novel application of active learning to optimize spectroscopic follow up for large scale astronomical surveys. Finally, much of this work requires learning functions over sets which we know must satisfy certain properties (e.g., permutation invariance). We conclude the thesis by presenting a novel neural network architecture for predicting the astronomical class of individual objects in the same exposure using a neural architecture specifically designed to accommodate known inductive biases present in the data

Design Amortization for Bayesian Optimal Experimental Design

Bayesian optimal experimental design is a sub-field of statistics focused on developing methods to make efficient use of experimental resources. Any potential design is evaluated in terms of a utility function, such as the (theoretically well-justified) expected information gain (EIG); unfortunately however, under most circumstances the EIG is intractable to evaluate. In this work we build off of successful variational approaches, which optimize a parameterized variational model with respect to bounds on the EIG. Past work focused on learning a new variational model from scratch for each new design considered. Here we present a novel neural architecture that allows experimenters to optimize a single variational model that can estimate the EIG for potentially infinitely many designs. To further improve computational efficiency, we also propose to train the variational model on a significantly cheaper-to-evaluate lower bound, and show empirically that the resulting model provides an excellent guide for more accurate, but expensive to evaluate bounds on the EIG. We demonstrate the effectiveness of our technique on generalized linear models, a class of statistical models that is widely used in the analysis of controlled experiments. Experiments show that our method is able to greatly improve accuracy over existing approximation strategies, and achieve these results with far better sample efficiency

How Have Astronomers Cited Other Fields in the Last Decade?

International audienceWe present a citation pattern analysis between astronomical papers and 13 other disciplines, based on the arXiv database over the past decade (2010–2020). We analyze 12,600 astronomical papers citing over 14,531 unique publications outside astronomy. Two striking patterns are unraveled. First, general relativity recently became the most cited field by astronomers, a trend highly correlated with the discovery of gravitational waves. Second, the fast growth of referenced papers in computer science and statistics, the first with a notable 15 fold increase since 2015. Such findings confirm the critical role of interdisciplinary efforts involving astronomy, statistics, and computer science in recent astronomical research

Are classification metrics good proxies for SN Ia cosmological constraining power?

International audienceContext: When selecting a classifier to use for a supernova Ia (SN Ia) cosmological analysis, it is common to make decisions based on metrics of classification performance, i.e. contamination within the photometrically classified SN Ia sample, rather than a measure of cosmological constraining power. If the former is an appropriate proxy for the latter, this practice would save those designing an analysis pipeline from the computational expense of a full cosmology forecast. Aims: This study tests the assumption that classification metrics are an appropriate proxy for cosmology metrics. Methods: We emulate photometric SN Ia cosmology samples with controlled contamination rates of individual contaminant classes and evaluate each of them under a set of classification metrics. We then derive cosmological parameter constraints from all samples under two common analysis approaches and quantify the impact of contamination by each contaminant class on the resulting cosmological parameter estimates. Results: We observe that cosmology metrics are sensitive to both the contamination rate and the class of the contaminating population, whereas the classification metrics are insensitive to the latter. Conclusions: We therefore discourage exclusive reliance on classification-based metrics for cosmological analysis design decisions, e.g. classifier choice, and instead recommend optimizing using a metric of cosmological parameter constraining power

Are classification metrics good proxies for SN Ia cosmological constraining power?

International audienceContext: When selecting a classifier to use for a supernova Ia (SN Ia) cosmological analysis, it is common to make decisions based on metrics of classification performance, i.e. contamination within the photometrically classified SN Ia sample, rather than a measure of cosmological constraining power. If the former is an appropriate proxy for the latter, this practice would save those designing an analysis pipeline from the computational expense of a full cosmology forecast. Aims: This study tests the assumption that classification metrics are an appropriate proxy for cosmology metrics. Methods: We emulate photometric SN Ia cosmology samples with controlled contamination rates of individual contaminant classes and evaluate each of them under a set of classification metrics. We then derive cosmological parameter constraints from all samples under two common analysis approaches and quantify the impact of contamination by each contaminant class on the resulting cosmological parameter estimates. Results: We observe that cosmology metrics are sensitive to both the contamination rate and the class of the contaminating population, whereas the classification metrics are insensitive to the latter. Conclusions: We therefore discourage exclusive reliance on classification-based metrics for cosmological analysis design decisions, e.g. classifier choice, and instead recommend optimizing using a metric of cosmological parameter constraining power

The DESI experiment part I: science, targeting, and survey design

DESI (Dark Energy Spectroscopic Instrument) is a Stage IV ground-based dark energy experiment that will study baryon acoustic oscillations (BAO) and the growth of structure through redshift-space distortions with a wide-area galaxy and quasar redshift survey. To trace the underlying dark matter distribution, spectroscopic targets will be selected in four classes from imaging data. We will measure luminous red galaxies up to $z=1.0$. To probe the Universe out to even higher redshift, DESI will target bright [O II] emission line galaxies up to $z=1.7$. Quasars will be targeted both as direct tracers of the underlying dark matter distribution and, at higher redshifts ($2.1 < z < 3.5$), for the Ly-$\alpha$ forest absorption features in their spectra, which will be used to trace the distribution of neutral hydrogen. When moonlight prevents efficient observations of the faint targets of the baseline survey, DESI will conduct a magnitude-limited Bright Galaxy Survey comprising approximately 10 million galaxies with a median $z\approx 0.2$. In total, more than 30 million galaxy and quasar redshifts will be obtained to measure the BAO feature and determine the matter power spectrum, including redshift space distortions

The DESI Experiment Part II: Instrument Design

DESI (Dark Energy Spectropic Instrument) is a Stage IV ground-based dark energy experiment that will study baryon acoustic oscillations and the growth of structure through redshift-space distortions with a wide-area galaxy and quasar redshift survey. The DESI instrument is a robotically-actuated, fiber-fed spectrograph capable of taking up to 5,000 simultaneous spectra over a wavelength range from 360 nm to 980 nm. The fibers feed ten three-arm spectrographs with resolution $R= \lambda/\Delta\lambda$ between 2000 and 5500, depending on wavelength. The DESI instrument will be used to conduct a five-year survey designed to cover 14,000 deg$^2$. This powerful instrument will be installed at prime focus on the 4-m Mayall telescope in Kitt Peak, Arizona, along with a new optical corrector, which will provide a three-degree diameter field of view. The DESI collaboration will also deliver a spectroscopic pipeline and data management system to reduce and archive all data for eventual public use