Massive data compression for parameter-dependent covariance matrices

We show how the massive data compression algorithm MOPED can be used to reduce, by orders of magnitude, the number of simulated data sets which are required to estimate the covariance matrix required for the analysis of Gaussian-distributed data. This is relevant when the covariance matrix cannot be calculated directly. The compression is especially valuable when the covariance matrix varies with the model parameters. In this case, it may be prohibitively expensive to run enough simulations to estimate the full covariance matrix throughout the parameter space. This compression may be particularly valuable for the next generation of weak lensing surveys, such as proposed for Euclid and Large Synoptic Survey Telescope, for which the number of summary data (such as band power or shear correlation estimates) is very large, ∼104, due to the large number of tomographic redshift bins which the data will be divided into. In the pessimistic case where the covariance matrix is estimated separately for all points in an Monte Carlo Markov Chain analysis, this may require an unfeasible 109 simulations. We show here that MOPED can reduce this number by a factor of 1000, or a factor of ∼106 if some regularity in the covariance matrix is assumed, reducing the number of simulations required to a manageable 103, making an otherwise intractable analysis feasible

De-noising of galaxy optical spectra with autoencoders

Optical spectra contain a wealth of information about the physical properties and formation histories of galaxies. Often though, spectra are too noisy for this information to be accurately retrieved. In this study, we explore how machine learning methods can be used to de-noise spectra and increase the amount of information we can gain without having to turn to sample averaging methods such as spectral stacking. Using machine learning methods trained on noise-added spectra - SDSS spectra with Gaussian noise added - we investigate methods of maximising the information we can gain from these spectra, in particular from emission lines, such that more detailed analysis can be performed. We produce a variational autoencoder (VAE) model, and apply it on a sample of noise-added spectra. Compared to the flux measured in the original SDSS spectra, the model values are accurate within 0.3-0.5 dex, depending on the specific spectral line and S/N. Overall, the VAE performs better than a principle component analysis (PCA) method, in terms of reconstruction loss and accuracy of the recovered line fluxes. To demonstrate the applicability and usefulness of the method in the context of large optical spectroscopy surveys, we simulate a population of spectra with noise similar to that in galaxies at z = 0.1 observed by the Dark Energy Spectroscopic Instrument (DESI). We show that we can recover the shape and scatter of the MZR in this ‘DESI-like’ sample, in a way that is not possible without the VAE-assisted de-noising

De-noising of galaxy optical spectra with autoencoders

Optical spectra contain a wealth of information about the physical properties and formation histories of galaxies. Often though, spectra are too noisy for this information to be accurately retrieved. In this study, we explore how machine learning methods can be used to de-noise spectra and increase the amount of information we can gain without having to turn to sample averaging methods such as spectral stacking. Using machine learning methods trained on noise-added spectra - SDSS spectra with Gaussian noise added - we investigate methods of maximising the information we can gain from these spectra, in particular from emission lines, such that more detailed analysis can be performed. We produce a variational autoencoder (VAE) model, and apply it on a sample of noise-added spectra. Compared to the flux measured in the original SDSS spectra, the model values are accurate within 0.3-0.5 dex, depending on the specific spectral line and S/N. Overall, the VAE performs better than a principle component analysis (PCA) method, in terms of reconstruction loss and accuracy of the recovered line fluxes. To demonstrate the applicability and usefulness of the method in the context of large optical spectroscopy surveys, we simulate a population of spectra with noise similar to that in galaxies at $z = 0.1$ observed by the Dark Energy Spectroscopic Instrument (DESI). We show that we can recover the shape and scatter of the MZR in this "DESI-like" sample, in a way that is not possible without the VAE-assisted de-noising.Comment: 14 pages, 10 figures, 6 tables, accepted for publication in MNRA

TELEMAC model archive: Integrating open–source tools for the management and visualisation of model data

Water Qualit