CosmoHub: Interactive exploration and distribution of astronomical data on Hadoop

We present CosmoHub (https://cosmohub.pic.es), a web application based on Hadoop to perform interactive exploration and distribution of massive cosmological datasets. Recent Cosmology seeks to unveil the nature of both dark matter and dark energy mapping the large-scale structure of the Universe, through the analysis of massive amounts of astronomical data, progressively increasing during the last (and future) decades with the digitization and automation of the experimental techniques. CosmoHub, hosted and developed at the Port d'Informació Científica (PIC), provides support to a worldwide community of scientists, without requiring the end user to know any Structured Query Language (SQL). It is serving data of several large international collaborations such as the Euclid space mission, the Dark Energy Survey (DES), the Physics of the Accelerating Universe Survey (PAUS) and the Marenostrum Institut de Ciències de l'Espai (MICE) numerical simulations. While originally developed as a PostgreSQL relational database web frontend, this work describes the current version of CosmoHub, built on top of Apache Hive, which facilitates scalable reading, writing and managing huge datasets. As CosmoHub's datasets are seldomly modified, Hive it is a better fit. Over 60 TiB of cataloged information and 50×10 astronomical objects can be interactively explored using an integrated visualization tool which includes 1D histogram and 2D heatmap plots. In our current implementation, online exploration of datasets of 10 objects can be done in a timescale of tens of seconds. Users can also download customized subsets of data in standard formats generated in few minutes.CosmoHub has been partially funded through projects of the Spanish national program “Programa Estatal de I + D + i” of the Spanish government. The support of the ERDF fund is gratefully acknowledged

The PAU survey: close galaxy pairs identification and analysis

Galaxy pairs constitute the initial building blocks of galaxy evolution, which is driven through merger events and interactions. Thus, the analysis of these systems can be valuable in understanding galaxy evolution and studying structure formation. In this work, we present a new publicly available catalogue of close galaxy pairs identified using photometric redshifts provided by the Physics of the Accelerating Universe Survey (PAUS). To efficiently detect them, we take advantage of the high-precision photo−z (σ68 < 0.02) and apply an identification algorithm previously tested using simulated data. This algorithm considers the projected distance between the galaxies (rp < 50 kpc), the projected velocity difference (ΔV < 3500 km s−1) and an isolation criterion to obtain the pair sample. We applied this technique to the total sample of galaxies provided by PAUS and to a subset with high-quality redshift estimates. Finally, the most relevant result we achieved was determining the mean mass for several subsets of galaxy pairs selected according to their total luminosity, colour, and redshift, using galaxy–galaxy lensing estimates. For pairs selected from the total sample of PAUS with a mean r-band luminosity 1010.6 h−2 L⊙, we obtain a mean mass of M200 = 1012.2 h−1 M⊙, compatible with the mass–luminosity ratio derived for elliptical galaxies. We also study the mass-to-light ratio M/L as a function of the luminosity L and find a lower M/L (or steeper slope with L) for pairs than the one extrapolated from the measurements in groups and galaxy clusters.The PAU Survey is partially supported by MINECO under grants CSD2007-00060, AYA2015-71825, ESP2017-89838, PGC2018-094773, PGC2018-102021, PID2019-111317GB, SEV-2016-0588, SEV-2016-0597,MDM-2015-0509 and Juan de la Cierva fellowship and LACEGAL and EWC Marie Sklodowska-Curie grant numbers 734374 and 776247 with ERDF funds from the EU Horizon 2020 Programme, some of which include ERDF funds from the European Union. IEEC and IFAE are partially funded by the CERCA and Beatriu de Pinos program of the Generalitat de Catalunya. Funding for PAUS has also been provided by Durham University (via the ERC StG DEGAS-259586), ETH Zurich, Leiden University (via ERC StG ADULT-279396 and Netherlands Organisation for Scientific Research (NWO) Vici grant 639.043.512), University College London and from the European Union’s Horizon 2020 research and innovation programme under the grant agreement number 776247 EWC. This project has received funding from the European Union’s Horizon 2020 Research and Innovation Programme under the Marie Sklodowska-Curie grant agreement number 734374. This work was also partially supported by the Consejo Nacional de Investigaciones Científicas y Técnicas (CONICET, Argentina) and the Secretaría de Ciencia y Tecnología de la Universidad Nacional de Córdoba (SeCyT-UNC, Argentina). This work has been also partially supported by the Polish National Agency for Academic Exchange (Bekker grant BPN/BEK/2021/1/00298/DEC/1), the European Union’s Horizon 2020 Research and Innovation programme under the Marie Sklodowska-Curie grant agreement (number 754510). H. Hildebrandt is supported by a Heisenberg grant of the Deutsche Forschungsgemeinschaft (Hi 1495/5-1) as well as an ERC consolidator grant (number 770935). A. Wittje is supported by the DFG (SFB 1491).Peer reviewe

Cross-correlating spectroscopic and photometric galaxy surveys

En esta tesis se estudia el acotamiento en los paraámetros cosmología al combinar observaciones en catálogos de galaxias espectroscópicas y fotométricos. Los catálogos fotométricos miden las distorsiones de lente gravitacional débil (WL), mientras que los catalogos espectroscópicos, con más alta precisión en la información de redshift (o corriento al rojo), son ideales para el estudio de distorsiones espaciales de redshift (RSD). El análisis combinado se realiza únicamente com funciones de correlación angular, lo que simplifica el estudio, en particular en lo que respecta a la inclusión de la covarianza entre observables. El primer capítulo presenta un nuevo algoritmo para el cálculo eficaz de las correlaciones cruzadas de varios marcadores, incluidos WL en correlaciones cruzadas con bines estrechos. Estimar la función de correlación angular es particularmente costoso dado que el número de correlaciones cruzadas aumenta como O(n^2), donde $n$ es el número de bines en redshift. Más adelante, el capítulo estudia el efecto de aproximación de Limber, y RSD en el modelado de correlaciones auto y cruzadas. Para bines de redshift delgados, la aproximación de Limber deja de funcionar y no permite incorporar las correlaciones cruzadas. Al disminuir el ancho de los bines en redshift, crece la amplitude de correlacion y el efecto de RSD, lo que redundará en beneficio del acotamiento de parametros cosmológicos. Una tendencia interesante es la contribución de las oscilaciones acústicas de bariones (BAO) en la correlacion cruzadas entre bines de distitnto redshift. La separacion en redshift entre dos bines reduce la amplitud de las correlaciones en escalas pequeñas, lo que aumenta el contraste en el BAO. También estudiamos la relación señal-ruido de diferentes correlaciones cruzadas. El segundo Capítulo presenta un pronóstico de cotas en la historia de la expansión y del crecimiento cósmico, usando un catalogo spectroscópico y otro fotométrico ficticios de 14000 grados cuadrados cada uno. Cuando estos catalogos se sobrelapan en la misma region del cielo, encontramos mejores cotas en los parametros cosmológicos. Esto es debido a las correlaciones cruzadas adicionales entre catálogos y la reducción de la varianza en el muestreo (debida a la covarianza entre trazadores). En primer lugar mostramos un estudio por separado de la dependencia en el ancho de bin en redshift, en RSD, en BAO y en WL. Encontramos ganancias equivalentes a tener el 30% mas de área en los catalogos cuando estos se superponen en el cielo. Por último, analizamos el origen de esta moderada ganancia en el contexto de la literatura existente. Diferentes grupos han reportado que al solapar los catalogos o bien no encuentran ningún beneficio o bien encuentran grandes beneficios. Nosotros sugerimos que la covarianza entre observables y el uso de diferentes observables puede explicar estas diferencias. El sesgo (bias) en galaxias, relaciona las sobredensidades de galaxias con las del campo de fluctuaciones de materia, de manera que la incertidumbre en el bias limita las predicciones. Por ello investigamos con detalle como las correlaciones cruzadas, RSD, BAO y WL afectan las medidas del bias en galaxias. En particular, cuando los catalogos sobrelapan disminuyen los errores en el bias para la muestras fotométrica. La última seccion cuantifica los beneficios de los "priors'' y los efectos de la estocasticidad en el bias. El impacto de las incertidumbres en las estocasticidad es menor cuando hay sobrelapamiento.In this thesis we study constraining cosmology when combining spectroscopic and photometric galaxy survey. The photometric survey measures galaxy shape distortions from Weak Lensing (WL), while high precision redshift information makes spectroscopic surveys ideal for redshift space distortions (RSD). The combined analysis is performed entirely in angular-correlation functions, which simplifies the joined analysis, in particular the inclusion of covariance between then. The first chapter introduce a novel algorithm for efficiently calculating the cross-correlations of multiple tracers (i.e. galaxy types/luminosities) and including WL in narrow redshift bin cross-correlations. Estimating the angular-correlations function is in particular demanding since the number for cross-correlations increase O(n^2) with $n$ being the number of redshift bins. Later the chapter study the effect of Limber approximation and RSD on the modeling of auto- and cross-correlations. For thin redshift bins, the Limber approximation completely breaks down and does not allow cross-correlations between redshift bins. Decreasing the bin width increases the amplitude of the galaxy correlations and the effect of RSD, which will benefit the cosmological constraints. One interesting trend is the baryon acoustic oscillation (BAO) contribution in the cross-correlations of redshift bins. The redshift separations between two bins reduce small-scale clustering, hence increasing the BAO contrast. We also study the signal-to-noise of different cross-correlations. The second chapter forecast the constraints on the cosmic expansion and growth history, using two fiducial 14000 sq deg. spectroscopic and photometric galaxy surveys. Overlapping surveys (same sky) has improved constraints from additional cross-correlations and sample variance cancellations (covariance in multiple tracers). We study first separate how redshift bin width, RSD, BAO and WL affect the forecast. We find gains equivalent to 30\% larger areas when using overlapping surveys. Last, we discuss the origen of this moderate gain in the context of existing literature. Different groups reports either none or high benefits for overlapping galaxy surveys. We suggest the covariance between surveys and different same-sky definitions (i.e. different observables) can explain the differences. Galaxy bias relate the galaxy overdensities to the underlying matter fluctuations, and the uncertainty in galaxy bias strongly affects the forecast. We therefore investigate in detail how cross-correlations, RSD, BAO and WL affects constraints on galaxy bias. Overlapping surveys in particular increase constraint on the bias from the photometric sample. Last section quantify the benefit of priors and the effect of bias stochasticity. The impact of uncertainties in bias stochasticity is less for overlapping surveys

Cross-correlating spectroscopic and photometric galaxy surveys

En esta tesis se estudia el acotamiento en los paraámetros cosmología al combinar observaciones en catálogos de galaxias espectroscópicas y fotométricos. Los catálogos fotométricos miden las distorsiones de lente gravitacional débil (WL), mientras que los catalogos espectroscópicos, con más alta precisión en la información de redshift (o corriento al rojo), son ideales para el estudio de distorsiones espaciales de redshift (RSD). El análisis combinado se realiza únicamente com funciones de correlación angular, lo que simplifica el estudio, en particular en lo que respecta a la inclusión de la covarianza entre observables. El primer capítulo presenta un nuevo algoritmo para el cálculo eficaz de las correlaciones cruzadas de varios marcadores, incluidos WL en correlaciones cruzadas con bines estrechos. Estimar la función de correlación angular es particularmente costoso dado que el número de correlaciones cruzadas aumenta como O(n^2), donde n es el número de bines en redshift. Más adelante, el capítulo estudia el efecto de aproximación de Limber, y RSD en el modelado de correlaciones auto y cruzadas. Para bines de redshift delgados, la aproximación de Limber deja de funcionar y no permite incorporar las correlaciones cruzadas. Al disminuir el ancho de los bines en redshift, crece la amplitude de correlacion y el efecto de RSD, lo que redundará en beneficio del acotamiento de parametros cosmológicos. Una tendencia interesante es la contribución de las oscilaciones acústicas de bariones (BAO) en la correlacion cruzadas entre bines de distitnto redshift. La separacion en redshift entre dos bines reduce la amplitud de las correlaciones en escalas pequeñas, lo que aumenta el contraste en el BAO. También estudiamos la relación señal-ruido de diferentes correlaciones cruzadas. El segundo Capítulo presenta un pronóstico de cotas en la historia de la expansión y del crecimiento cósmico, usando un catalogo spectroscópico y otro fotométrico ficticios de 14000 grados cuadrados cada uno. Cuando estos catalogos se sobrelapan en la misma region del cielo, encontramos mejores cotas en los parametros cosmológicos. Esto es debido a las correlaciones cruzadas adicionales entre catálogos y la reducción de la varianza en el muestreo (debida a la covarianza entre trazadores). En primer lugar mostramos un estudio por separado de la dependencia en el ancho de bin en redshift, en RSD, en BAO y en WL. Encontramos ganancias equivalentes a tener el 30% mas de área en los catalogos cuando estos se superponen en el cielo. Por último, analizamos el origen de esta moderada ganancia en el contexto de la literatura existente. Diferentes grupos han reportado que al solapar los catalogos o bien no encuentran ningún beneficio o bien encuentran grandes beneficios. Nosotros sugerimos que la covarianza entre observables y el uso de diferentes observables puede explicar estas diferencias. El sesgo (bias) en galaxias, relaciona las sobredensidades de galaxias con las del campo de fluctuaciones de materia, de manera que la incertidumbre en el bias limita las predicciones. Por ello investigamos con detalle como las correlaciones cruzadas, RSD, BAO y WL afectan las medidas del bias en galaxias. En particular, cuando los catalogos sobrelapan disminuyen los errores en el bias para la muestras fotométrica. La última seccion cuantifica los beneficios de los "priors'' y los efectos de la estocasticidad en el bias. El impacto de las incertidumbres en las estocasticidad es menor cuando hay sobrelapamiento.In this thesis we study constraining cosmology when combining spectroscopic and photometric galaxy survey. The photometric survey measures galaxy shape distortions from Weak Lensing (WL), while high precision redshift information makes spectroscopic surveys ideal for redshift space distortions (RSD). The combined analysis is performed entirely in angular-correlation functions, which simplifies the joined analysis, in particular the inclusion of covariance between then. The first chapter introduce a novel algorithm for efficiently calculating the cross-correlations of multiple tracers (i.e. galaxy types/luminosities) and including WL in narrow redshift bin cross-correlations. Estimating the angular-correlations function is in particular demanding since the number for cross-correlations increase O(n^2) with n being the number of redshift bins. Later the chapter study the effect of Limber approximation and RSD on the modeling of auto- and cross-correlations. For thin redshift bins, the Limber approximation completely breaks down and does not allow cross-correlations between redshift bins. Decreasing the bin width increases the amplitude of the galaxy correlations and the effect of RSD, which will benefit the cosmological constraints. One interesting trend is the baryon acoustic oscillation (BAO) contribution in the cross-correlations of redshift bins. The redshift separations between two bins reduce small-scale clustering, hence increasing the BAO contrast. We also study the signal-to-noise of different cross-correlations. The second chapter forecast the constraints on the cosmic expansion and growth history, using two fiducial 14000 sq deg. spectroscopic and photometric galaxy surveys. Overlapping surveys (same sky) has improved constraints from additional cross-correlations and sample variance cancellations (covariance in multiple tracers). We study first separate how redshift bin width, RSD, BAO and WL affect the forecast. We find gains equivalent to 30\% larger areas when using overlapping surveys. Last, we discuss the origen of this moderate gain in the context of existing literature. Different groups reports either none or high benefits for overlapping galaxy surveys. We suggest the covariance between surveys and different same-sky definitions (i.e. different observables) can explain the differences. Galaxy bias relate the galaxy overdensities to the underlying matter fluctuations, and the uncertainty in galaxy bias strongly affects the forecast. We therefore investigate in detail how cross-correlations, RSD, BAO and WL affects constraints on galaxy bias. Overlapping surveys in particular increase constraint on the bias from the photometric sample. Last section quantify the benefit of priors and the effect of bias stochasticity. The impact of uncertainties in bias stochasticity is less for overlapping surveys

The PAU Survey: background light estimation with deep learning techniques

In any imaging survey, measuring accurately the astronomical background light is crucial to obtain good photometry. This paper introduces BKGNET, a deep neural network to predict the background and its associated error. BKGNET has been developed for data from the Physics of the Accelerating Universe Survey (PAUS), an imaging survey using a 40 narrow-band filter camera (PAUCam). The images obtained with PAUCam are affected by scattered light: an optical effect consisting of light multiply reflected that deposits energy in specific detector regions affecting the science measurements. Fortunately, scattered light is not a random effect, but it can be predicted and corrected for. We have found that BKGNET background predictions are very robust to distorting effects, while still being statistically accurate. On average, the use of BKGnet improves the photometric flux measurements by 7 per cent and up to 20 per cent at the bright end. BKGNET also removes a systematic trend in the background error estimation with magnitude in the i band that is present with the current PAU data management method. With BKGNET, we reduce the photometric redshift outlier rate by 35 per cent for the best 20 per cent galaxies selected with a photometric quality parameter.Funding for PAUS has been provided by Durham University (via the ERC StG DEGAS-259586), ETH Zurich, Leiden University (via ERC StG ADULT-279396 and Netherlands Organisation for Scientific Research (NWO) Vici grant 639.043.512) and University College London. The PAUS participants from Spanish institutions are partially supported by MINECO under grants CSD2007-00060, AYA2015-71825, ESP2015-88861, FPA2015-68048, SEV-2016-0588, SEV-2016-0597, and MDM-2015-0509, some of which include ERDF funds from the European Union. IEEC and IFAE are partially funded by the CERCA program of the Generalitat de Catalunya. The PAU data center is hosted by the Port d’Informacio Cientifica (PIC), maintained through a collaboration of CIEMAT and IFAE, with additional support from Universitat Autonoma de Barcelona and ERDF. CosmoHub has been developed by PIC and was partially funded by the ‘Plan Estatal de Investigacion Cientifica y Tecnica y de Innovacion program of the Spanish government. We gratefully acknowledge the support of NVIDIA Corporation with the donation of the Titan V GPU used for this research. This project has received funding from the European Union’s Horizon 2020 research and innovation programme under grant agreement No 776247. AA is supported by a Royal Society Wolfson Fellowship. MS has been supported by the National Science Centre (grant UMO2016/23/N/ST9/02963).Peer reviewe