High performance computing of massive Astrometry and Photometry data from Gaia

[spa] Gaia es la misión espacial astrométrica más ambiciosa de la Agencia Espacial Europea (ESA). El satélite fue lanzado el 19 de Diciembre de 2013 y su objetivo principal es la determinación, con una resolución y precisión sin precedentes, de las posiciones, distancias y velocidades de más de mil millones de estrellas de nuestra galaxia. Esta Tesis se centra en el desarrollo del sistema de procesado IDU, "Intermediate Data Updating". IDU es una de las etapas de calibración instrumental y reducción de datos astrométricos más exigente del sistema de procesado del satélite Gaia. Sin este sistema, Gaia no podría alcanzar el nivel de precisión que se quiere obtener y su presencia es fundamental para lograr la convergencia óptima del sistema iterativo de procesado de datos en el que se basa la reducción de datos de Gaia. El procesado de los datos de Gaia es un gran reto tecnológico. En particular, el gran volumen de datos a procesar y el elevado número de procesos involucrados ha implicado el diseño de un sistema de distribución y procesado de datos muy complejo. Este procesado se basa en un sistema iterativo entre varios procesos en el que se añaden de manera continuada los nuevos datos recibidos del satélite. De entre estos procesos, esta tesis se centra en el diseño e implementación de IDU, donde se vuelven a procesar todos los datos brutos usando las calibraciones más recientes obtenidas del resto de procesos. El diseño e implementación de IDU ha supuesto una gran variedad de retos; incluyendo los problemas puramente científicos pero también las dificultades técnicas que aparecen en el procesado del gran volumen de datos de Gaia y la gestión de todas las tareas de desarrollo, test y coordinación de los equipos que contribuyen a este sistema. IDU se ejecuta en el supercomputador Marenostrum, gestionado por el "Barcelona Supercomputing Center" (BSC). Finalmente, esta tesis incluye los resultados de la primera ejecución operacional de IDU la cual ha servido para demostrar que el sistema desarrollado esta listo para afrontar el exigente reto de procesar los datos reales de Gaia durante los próximos años de misión.[eng] Gaia is an extremely ambitious astrometric space mission adopted within the scientific programme of the European Space Agency (ESA) in October 2000. It aims to measure with very high accuracy the positions, motions and parallaxes of a large number of stars and galactic objects, including also for almost all the objects information about their brightness, colour, radial velocity, orbits and astrophysical parameters. Gaia requires a demanding data processing system on both data volume and processing power. The treatment of the Gaia data has been designed as an iterative process between several systems each one solving different aspects of the data reduction system. In this thesis we have addressed the design and implementation of the Intermediate Data Updating (IDU) system. IDU is the instrument calibration and astrometric data processing system more demanding in data volume and processing power of the data processing system of the Gaia satellite data. Without this system, Gaia would not be able to provide the envisaged accuracies and its presence is fundamental to get the optimum convergence of the iterative process on which all the data processing of the spacecraft is based. The design and implementation of an efficient IDU system is not a simple task and a good knowledge of the Gaia mission is fundamental. The design and implementation of IDU is not only referring to the actual design and coding of the system but also to the management and scheduling of all the related development tasks, system tests and in addition the coordination of the teams contributing to this system. The developed system is very flexible and modular so it can be easily adapted and extended to cope with the changes on the operational processing requirements. In addition, the design and implementation of IDU presents a variety of interesting challenges; covering not only the purely scientific problems that appear in any data reduction but also the technical issues for the processing of the huge amount of data that Gaia is providing. The design has also been driven by the characteristics and restrictions of the execution environment and resources -- Marenostrum supercomputer hosted by the Barcelona Supercomputing Center (BSC) (Spain). Furthermore, we have developed several tools to make the handling of the data easier; including tailored data access routines, efficient data formats and an autonomous application in charge of handling and checking the correctness of all the input data entering and produced by IDU. Finally, we have been able to test and demonstrate how all the work done in the design and implementation of IDU is more than capable of dealing with the real Gaia data processing. We have basically executed two of the IDU tasks over the first ten months of routine operational Gaia data. This execution has been the very first cyclic data processing level run over real data so far. Executing IDU at Marenostrum over that amount of data for the first time has been a challenging task and from the results obtained we are confident that the system, we have designed and that constitutes the bulk of this thesis, is ready to cope with the Gaia data according to the requirements sets. Furthermore, the presented design provides a solid IDU system foundation for the challenging task of processing the Gaia data during the forthcoming years

Gaia Focused Product Release: Radial velocity time series of long-period variables

Context. The third Gaia Data Release (DR3) provided photometric time series of more than 2 million long-period variable (LPV) candidates. Anticipating the publication of full radial-velocity data planned with Data Release 4, this Focused Product Release (FPR) provides radial-velocity time series for a selection of LPV candidates with high-quality observations. Aims. We describe the production and content of the Gaia catalog of LPV radial-velocity time series, and the methods used to compute the variability parameters published as part of the Gaia FPR. Methods. Starting from the DR3 catalog of LPV candidates, we applied several filters to construct a sample of sources with high-quality radial-velocity measurements. We modeled their radial-velocity and photometric time series to derive their periods and amplitudes, and further refined the sample by requiring compatibility between the radial-velocity period and at least one of the G, GBP, or GRP photometric periods. Results. The catalog includes radial-velocity time series and variability parameters for 9614 sources in the magnitude range 6 ≲ G/mag ≲ 14, including a flagged top-quality subsample of 6093 stars whose radial-velocity periods are fully compatible with the values derived from the G, GBP, and GRP photometric time series. The radial-velocity time series contain a mean of 24 measurements per source taken unevenly over a duration of about three years. We identify the great majority of the sources (88%) as genuine LPV candidates, with about half of them showing a pulsation period and the other half displaying a long secondary period. The remaining 12% of the catalog consists of candidate ellipsoidal binaries. Quality checks against radial velocities available in the literature show excellent agreement. We provide some illustrative examples and cautionary remarks. Conclusions. The publication of radial-velocity time series for almost ten thousand LPV candidates constitutes, by far, the largest such database available to date in the literature. The availability of simultaneous photometric measurements gives a unique added value to the Gaia catalog

Gaia Data Release 1. Astrometry: one billion positions, two million proper motions and parallaxes

Context. Gaia Data Release 1 (DR1) contains astrometric results for more than 1 billion stars brighter than magnitude 20.7 based on observations collected by the Gaia satellite during the first 14 months of its operational phase. Aims. We give a brief overview of the astrometric content of the data release and of the model assumptions, data processing, and validation of the results. Methods. For stars in common with the Hipparcos and Tycho-2 catalogues, complete astrometric single-star solutions are obtained by incorporating positional information from the earlier catalogues. For other stars only their positions are obtained, essentially by neglecting their proper motions and parallaxes. The results are validated by an analysis of the residuals, through special validation runs, and by comparison with external data. Results. For about two million of the brighter stars (down to magnitude ∼11.5) we obtain positions, parallaxes, and proper motions to Hipparcos-type precision or better. For these stars, systematic errors depending for example on position and colour are at a level of ±0.3 milliarcsecond (mas). For the remaining stars we obtain positions at epoch J2015.0 accurate to ∼10 mas. Positions and proper motions are given in a reference frame that is aligned with the International Celestial Reference Frame (ICRF) to better than 0.1 mas at epoch J2015.0, and non-rotating with respect to ICRF to within 0.03 mas yr−1 . The Hipparcos reference frame is found to rotate with respect to the Gaia DR1 frame at a rate of 0.24 mas yr−1 . Conclusions. Based on less than a quarter of the nominal mission length and on very provisional and incomplete calibrations, the quality and completeness of the astrometric data in Gaia DR1 are far from what is expected for the final mission products. The present results nevertheless represent a huge improvement in the available fundamental stellar data and practical definition of the optical reference frame

Gaia Data Release 2 - The astrometric solution

Context. Gaia Data Release 2 (Gaia DR2) contains results for 1693 million sources in the magnitude range 3 to 21 based on observations collected by the European Space Agency Gaia satellite during the first 22 months of its operational phase. Aims. We describe the input data, models, and processing used for the astrometric content of Gaia DR2, and the validation of these results performed within the astrometry task. Methods. Some 320 billion centroid positions from the pre-processed astrometric CCD observations were used to estimate the five astrometric parameters (positions, parallaxes, and proper motions) for 1332 million sources, and approximate positions at the reference epoch J2015.5 for an additional 361 million mostly faint sources. These data were calculated in two steps. First, the satellite attitude and the astrometric calibration parameters of the CCDs were obtained in an astrometric global iterative solution for 16 million selected sources, using about 1% of the input data. This primary solution was tied to the extragalactic International Celestial Reference System (ICRS) by means of quasars. The resulting attitude and calibration were then used to calculate the astrometric parameters of all the sources. Special validation solutions were used to characterise the random and systematic errors in parallax and proper motion. Results. For the sources with five-parameter astrometric solutions, the median uncertainty in parallax and position at the reference epoch J2015.5 is about 0.04 mas for bright (G < 14 mag) sources, 0.1 mas at G = 17 mag, and 0.7 mas at G = 20 mag. In the proper motion components the corresponding uncertainties are 0.05, 0.2, and 1.2 mas yr 1, respectively. The optical reference frame defined by Gaia DR2 is aligned with ICRS and is non-rotating with respect to the quasars to within 0.15 mas yr 1. From the quasars and validation solutions we estimate that systematics in the parallaxes depending on position, magnitude, and colour are generally below 0.1 mas, but the parallaxes are on the whole too small by about 0.03 mas. Significant spatial correlations of up to 0.04 mas in parallax and 0.07 mas yr 1 in proper motion are seen on small (<1 deg) and intermediate (20 deg) angular scales. Important statistics and information for the users of the Gaia DR2 astrometry are given in the appendices

Gaia Focused Product Release: Sources from Service Interface Function image analysis

Context. Gaia’s readout window strategy is challenged by very dense fields in the sky. Therefore, in addition to standard Gaia observations, full Sky Mapper (SM) images were recorded for nine selected regions in the sky. A new software pipeline exploits these Service Interface Function (SIF) images of crowded fields (CFs), making use of the availability of the full two-dimensional (2D) information. This new pipeline produced half a million additional Gaia sources in the region of the omega Centauri (ω Cen) cluster, which are published with this Focused Product Release. We discuss the dedicated SIF CF data reduction pipeline, validate its data products, and introduce their Gaia archive table. Aims. Our aim is to improve the completeness of the Gaia source inventory in a very dense region in the sky, ω Cen. Methods. An adapted version of Gaia’s Source Detection and Image Parameter Determination software located sources in the 2D SIF CF images. These source detections were clustered and assigned to new SIF CF or existing Gaia sources by Gaia s cross-match software. For the new sources, astrometry was calculated using the Astrometric Global Iterative Solution software, and photometry was obtained in the Gaia DR3 reference system. We validated the results by comparing them to the public Gaia DR3 catalogue and external Hubble Space Telescope data. Results. With this Focused Product Release, 526 587 new sources have been added to the Gaia catalogue in ω Cen. Apart from positions and brightnesses, the additional catalogue contains parallaxes and proper motions, but no meaningful colour information. While SIF CF source parameters generally have a lower precision than nominal Gaia sources, in the cluster centre they increase the depth of the combined catalogue by three magnitudes and improve the source density by a factor of ten. Conclusions. This first SIF CF data publication already adds great value to the Gaia catalogue. It demonstrates what to expect for the fourth Gaia catalogue, which will contain additional sources for all nine SIF CF regions

Gaia Data Release 1. Open cluster astrometry: performance, limitations, and future prospects

Context. The first Gaia Data Release contains the Tycho-Gaia Astrometric Solution (TGAS). This is a subset of about 2 million stars for which, besides the position and photometry, the proper motion and parallax are calculated using Hipparcos and Tycho-2 positions in 1991.25 as prior information. Aims. We investigate the scientific potential and limitations of the TGAS component by means of the astrometric data for open clusters. Methods. Mean cluster parallax and proper motion values are derived taking into account the error correlations within the astrometric solutions for individual stars, an estimate of the internal velocity dispersion in the cluster, and, where relevant, the e ects of the depth of the cluster along the line of sight. Internal consistency of the TGAS data is assessed. Results. Values given for standard uncertainties are still inaccurate and may lead to unrealistic unit-weight standard deviations of least squares solutions for cluster parameters. Reconstructed mean cluster parallax and proper motion values are generally in very good agreement with earlier Hipparcos-based determination, although the Gaia mean parallax for the Pleiades is a significant exception. We have no current explanation for that discrepancy. Most clusters are observed to extend to nearly 15 pc from the cluster centre, and it will be up to future Gaia releases to establish whether those potential cluster-member stars are still dynamically bound to the clusters. Conclusions. The Gaia DR1 provides the means to examine open clusters far beyond their more easily visible cores, and can provide membership assessments based on proper motions and parallaxes. A combined HR diagram shows the same features as observed before using the Hipparcos data, with clearly increased luminosities for older A and F dwarfs

Gaia Data Release 1. Open cluster astrometry: performance, limitations, and future prospects

Context. The first Gaia Data Release contains the Tycho-Gaia Astrometric Solution (TGAS). This is a subset of about 2 million stars for which, besides the position and photometry, the proper motion and parallax are calculated using Hipparcos and Tycho-2 positions in 1991.25 as prior information. Aims. We investigate the scientific potential and limitations of the TGAS component by means of the astrometric data for open clusters. Methods. Mean cluster parallax and proper motion values are derived taking into account the error correlations within the astrometric solutions for individual stars, an estimate of the internal velocity dispersion in the cluster, and, where relevant, the e ects of the depth of the cluster along the line of sight. Internal consistency of the TGAS data is assessed. Results. Values given for standard uncertainties are still inaccurate and may lead to unrealistic unit-weight standard deviations of least squares solutions for cluster parameters. Reconstructed mean cluster parallax and proper motion values are generally in very good agreement with earlier Hipparcos-based determination, although the Gaia mean parallax for the Pleiades is a significant exception. We have no current explanation for that discrepancy. Most clusters are observed to extend to nearly 15 pc from the cluster centre, and it will be up to future Gaia releases to establish whether those potential cluster-member stars are still dynamically bound to the clusters. Conclusions. The Gaia DR1 provides the means to examine open clusters far beyond their more easily visible cores, and can provide membership assessments based on proper motions and parallaxes. A combined HR diagram shows the same features as observed before using the Hipparcos data, with clearly increased luminosities for older A and F dwarfs

Gaia Data Release 2: Summary of the contents and survey properties

Context. We present the second Gaiadata release, Gaia DR2, consisting of astrometry, photometry, radial velocities, and information on astrophysical parameters and variability, for sources brighter than magnitude 21. In addition epoch astrometry and photometry are provided for a modest sample of minor planets in the solar system. Aims. A summary of the contents of Gaia DR2 is presented, accompanied by a discussion on the differences with respect to Gaia DR1 and an overview of the main limitations which are still present in the survey. Recommendations are made on the responsible use of Gaia DR2 results. Methods. The raw data collected with the Gaia instruments during the first 22 months of the mission have been processed by the Gaia Data Processing and Analysis Consortium (DPAC) and turned into this second data release, which represents a major advance with respect to Gaia DR1 in terms of completeness, performance, and richness of the data products. Results. Gaia DR2 contains celestial positions and the apparent brightness in G for approximately 1.7 billion sources. For 1.3 billion of those sources, parallaxes and proper motions are in addition available. The sample of sources for which variability information is provided is expanded to 0.5 million stars. This data release contains four new elements: broad-band colour information in the form of the apparent brightness in the GBP (330-680 nm) and GRP (630-1050 nm) bands is available for 1.4 billion sources; median radial velocities for some 7 million sources are presented; for between 77 and 161 million sources estimates are provided of the stellar effective temperature, extinction, reddening, and radius and luminosity; and for a pre-selected list of 14 000 minor planets in the solar system epoch astrometry and photometry are presented. Finally, Gaia DR2 also represents a new materialisation of the celestial reference frame in the optical, the Gaia-CRF2, which is the first optical reference frame based solely on extragalactic sources. There are notable changes in the photometric system and the catalogue source list with respect to Gaia DR1, and we stress the need to consider the two data releases as independent. Conclusions. Gaia DR2 represents a major achievement for the Gaia mission, delivering on the long standing promise to provide parallaxes and proper motions for over 1 billion stars, and representing a first step in the availability of complementary radial velocity and source astrophysical information for a sample of stars in the Gaia survey which covers a very substantial fraction of the volume of our galaxy

Gaia Data Release 2, Variable stars in the colour-absolute magnitude diagram

Context: The ESA Gaia mission provides a unique time-domain survey for more than 1.6 billion sources with G<21 mag. Aims: We showcase stellar variability in the Galactic colour-absolute magnitude diagram (CaMD). We focus on pulsating, eruptive, and cataclysmic variables, as well as on stars that exhibit variability that is due to rotation and eclipses. Methods: We describe the locations of variable star classes, variable object fractions, and typical variability amplitudes throughout the CaMD and show how variability-related changes in colour and brightness induce 'motions'. To do this, we use 22 months of calibrated photometric, spectro-photometric, and astrometric Gaia data of stars with a significant parallax. To ensure that a large variety of variable star classes populate the CaMD, we crossmatched Gaia sources with known variable stars. We also used the statistics and variability detection modules of the Gaia variability pipeline. Corrections for interstellar extinction are not implemented in this article. Results: Gaia enables the first investigation of Galactic variable star populations in the CaMD on a similar, if not larger, scale as was previously done in the Magellanic Clouds. Although the observed colours are not corrected for reddening, distinct regions are visible in which variable stars occur. We determine variable star fractions to within the current detection thresholds of Gaia. Finally, we report the most complete description of variability-induced motion within the CaMD to date. Conclusions: Gaia enables novel insights into variability phenomena for an unprecedented number of stars, which will benefit the understanding of stellar astrophysics. The CaMD of Galactic variable stars provides crucial information on physical origins of variability in a way that has previously only been accessible for Galactic star clusters or external galaxies. Future Gaia data releases will enable significant improvements over this preview by providing longer time series, more accurate astrometry, and additional data types (time series BP and RP spectra, RVS spectra, and radial velocities), all for much larger samples of stars

Gaia Data Release 2: Observational Hertzsprung-Russell diagrams

Context. Gaia Data Release 2 provides high-precision astrometry and three-band photometry for about 1.3 billion sources over the full sky. The precision, accuracy, and homogeneity of both astrometry and photometry are unprecedented. Aims. We highlight the power of the Gaia DR2 in studying many fine structures of the Hertzsprung-Russell diagram (HRD). Gaia allows us to present many different HRDs, depending in particular on stellar population selections.We do not aim here for completeness in terms of types of stars or stellar evolutionary aspects. Instead, we have chosen several illustrative examples. Methods. We describe some of the selections that can be made in Gaia DR2 to highlight the main structures of the Gaia HRDs. We select both field and cluster (open and globular) stars, compare the observations with previous classifications and with stellar evolutionary tracks, and we present variations of the Gaia HRD with age, metallicity, and kinematics. Late stages of stellar evolution such as hot subdwarfs, post-AGB stars, planetary nebulae, and white dwarfs are also analysed, as well as low-mass brown dwarf objects. Results. The Gaia HRDs are unprecedented in both precision and coverage of the various Milky Way stellar populations and stellar evolutionary phases. Many fine structures of the HRDs are presented. The clear split of the white dwarf sequence into hydrogen and helium white dwarfs is presented for the first time in an HRD. The relation between kinematics and the HRD is nicely illustrated. Two different populations in a classical kinematic selection of the halo are unambiguously identified in the HRD. Membership and mean parameters for a selected list of open clusters are provided. They allow drawing very detailed cluster sequences, highlighting fine structures, and providing extremely precise empirical isochrones that will lead to more insight in stellar physics. Conclusions. Gaia DR2 demonstrates the potential of combining precise astrometry and photometry for large samples for studies in stellar evolution and stellar population and opens an entire new area for HRD-based studies