Interactive (statistical) visualisation and exploration of a billion objects with Vaex

With new catalogues arriving such as the Gaia DR1, containing more than a billion objects, new methods of handling and visualizing these data volumes are needed. In visualization, one problem is that the number of datapoints can become so large, that a scatter plot becomes cluttered. Another problem is that with over a billion objects, only a few cpu cycles are available per object if one wants to process them within a second, making traditional methods by rendering glyphs not viable. Instead, we show that by calculating statistics on a regular (N-dimensional) grid, visualizations of a billion objects can be done within a second on a modern desktop computer. This is achieved using memory mapping of hdf5 files together with a simple binning algorithm, which are part of a Python library called vaex. This enables efficient exploration or large datasets interactively, making science exploration of large catalogues feasible. Vaex is a Python library, which also integrates well in the Jupyter/Numpy/Astropy/matplotlib stack. Build on top of this is the vaex application, which allows for interactive exploration and visualization. The motivation for developing vaex is the catalogue of the Gaia satellite, however, vaex can also be used on SPH or N-body simulations, any other (future) catalogues such as SDSS, Pan-STARRS, LSST, WISE, 2MASS, etc. or other tabular data. The homepage for vaex is http://vaex.astro.rug.nl.Comment: 6 pages, 4 figures, conference proceeding for the IAU symposium 325 on Astroinformatics (accepted), webpage http://vaex.astro.rug.n

Vaex: Big Data exploration in the era of Gaia

We present a new Python library called vaex, to handle extremely large tabular datasets, such as astronomical catalogues like the Gaia catalogue, N-body simulations or any other regular datasets which can be structured in rows and columns. Fast computations of statistics on regular N-dimensional grids allows analysis and visualization in the order of a billion rows per second. We use streaming algorithms, memory mapped files and a zero memory copy policy to allow exploration of datasets larger than memory, e.g. out-of-core algorithms. Vaex allows arbitrary (mathematical) transformations using normal Python expressions and (a subset of) numpy functions which are lazily evaluated and computed when needed in small chunks, which avoids wasting of RAM. Boolean expressions (which are also lazily evaluated) can be used to explore subsets of the data, which we call selections. Vaex uses a similar DataFrame API as Pandas, a very popular library, which helps migration from Pandas. Visualization is one of the key points of vaex, and is done using binned statistics in 1d (e.g. histogram), in 2d (e.g. 2d histograms with colormapping) and 3d (using volume rendering). Vaex is split in in several packages: vaex-core for the computational part, vaex-viz for visualization mostly based on matplotlib, vaex-jupyter for visualization in the Jupyter notebook/lab based in IPyWidgets, vaex-server for the (optional) client-server communication, vaex-ui for the Qt based interface, vaex-hdf5 for hdf5 based memory mapped storage, vaex-astro for astronomy related selections, transformations and memory mapped (column based) fits storage. Vaex is open source and available under MIT license on github, documentation and other information can be found on the main website: https://vaex.io, https://docs.vaex.io or https://github.com/maartenbreddels/vaexComment: 14 pages, 8 figures, Submitted to A&A, interactive version of Fig 4: https://vaex.io/paper/fig

What can Gaia proper motions tell us about Milky Way dwarf galaxies?

We present a proper-motion study on models of the dwarf spheroidal galaxy Sculptor, based on the predicted proper-motion accuracy of Gaia measurements. Gaia will measure proper motions of several hundreds of stars for a Sculptor-like system. Even with an uncertainty on the proper motion of order 1.5 times the size of an individual proper-motion value of ~10 mas/century, we find that it is possible to recover Sculptor's systemic proper motion at its distance of 79 kpc.Comment: 5 pages, 1 figure; to appear in the proceedings of the GREAT-ITN conference on "The Milky Way Unravelled by Gaia", Barcelona (Dec 1-5 2014

Matching the dark matter profiles of dSph galaxies with those of simulated satellites: a two parameter comparison

We compare the dark matter halos' structural parameters derived for four Milky Way dwarf spheroidal galaxies to those of subhalos found in cosmological $N$-body simulations. We confirm that estimates of the mass at a single fixed radius are fully consistent with the observations. However, when a second structural parameter such as the logarithmic slope of the dark halo density profile measured close to the half-light radius is included in the comparison, we find little to no overlap between the satellites and the subhalos. Typically the right mass subhalos have steeper profiles at these radii than measurements of the dSph suggest. Using energy arguments we explore if it is possible to solve this discrepancy by invoking baryonic effects. Assuming that feedback from supernovae can lead to a reshaping of the halos, we compute the required efficiency and find entirely plausible values for a significant fraction of the subhalos and even as low as 0.1%. This implies that care must be taken not to exaggerate the effect of supernovae feedback as this could make the halos too shallow. These results could be used to calibrate and possibly constrain feedback recipes in hydrodynamical simulations.Comment: 6 pages, 3 figures, submitted to ApJ

Complexity on dwarf galaxies scale: A bimodal distribution function in Sculptor

In previous work we have presented Schwarzschild models of the Sculptor dSph, demonstrating that this system could be embedded in dark matter halos that are either cusped or cored. Here we show that the non-parametric distribution function recovered through Schwarschild's method is bimodal in energy and angular momentum space for all best fitting mass models explored. We demonstrate that this bimodality is directly related to the two components known to be present in Sculptor through stellar populations analysis, although our method is purely dynamical in nature and does not use this prior information. It therefore constitutes independent confirmation of the existence of two physically distinct dynamical components in Sculptor and suggests a rather complex assembly history for this dwarf galaxy.Comment: 4 pages, 4 figures, 1 table, accepted to ApJ Letter

Absolute Magnitude Calibration for Giants based on the Colour-Magnitude Diagrams of Galactic Clusters. II-Calibration with SDSS

We present an absolute magnitude calibration for red giants with the colour magnitude diagrams of six Galactic clusters with different metallicities i.e. M92, M13, M3, M71, NGC 6791 and NGC 2158. The combination of the absolute magnitudes of the red giant sequences with the corresponding metallicities provides calibration for absolute magnitude estimation for red giants for a given $(g-r)_{0}$ colour. The calibration is defined in the colour interval 0.45 $\leq(g-r)_{0}\leq$ 1.30 mag and it covers the metallicity interval $-2.15\leq \lbrack Fe/H \rbrack \leq$ +0.37 dex. The absolute magnitude residuals obtained by the application of the procedure to another set of Galactic clusters lie in the interval $-0.28< \Delta M \leq +0.43$ mag. However, the range of 94% of the residuals is shorter, $-0.1<\Delta M \leq+0.4$ mag. The mean and the standard deviation of (all) residuals are 0.169 and 0.140 mag, respectively. The derived relations are applicable to stars older than 2 Gyr, the age of the youngest calibrating cluster.Comment: 12 pages, including 5 figures and 10 tables, accepted for publication in PASA. arXiv admin note: substantial text overlap with arXiv:1204.429

A box full of chocolates: The rich structure of the nearby stellar halo revealed by Gaia and RAVE

The hierarchical structure formation model predicts that stellar halos should form, at least partly, via mergers. If this was a predominant formation channel for the Milky Way's halo, imprints of this merger history in the form of moving groups or streams should exist also in the vicinity of the Sun. Here we study the kinematics of halo stars in the Solar neighbourhood using the very recent first data release from the Gaia mission, and in particular the TGAS dataset, in combination with data from the RAVE survey. Our aim is to determine the amount of substructure present in the phase-space distribution of halo stars that could be linked to merger debris. To characterise kinematic substructure, we measure the velocity correlation function in our sample of halo (low metallicity) stars. We also study the distribution of these stars in the space of energy and two components of the angular momentum, in what we call "Integrals of Motion" space. The velocity correlation function reveals substructure in the form of an excess of pairs of stars with similar velocities, well above that expected for a smooth distribution. Comparison to cosmological simulations of the formation of stellar halos indicate that the levels found are consistent with the Galactic halo having been built fully via accretion. Similarly, the distribution of stars in the space of "Integrals of motion" is highly complex. A strikingly high fraction (between 58% and upto 73%) of the stars that are somewhat less bound than the Sun are on (highly) retrograde orbits. A simple comparison to Milky Way-mass galaxies in cosmological hydrodynamical simulations suggests that less than 1% have such prominently retrograde outer halos. We also identify several other statistically significant structures in "Integrals of Motion" space that could potentially be related to merger events.Comment: 19 pages, 16 figures. A&A in pres

Absolute Magnitude Calibration for Red Giants based on the Colour-Magnitude Diagrams of Galactic Clusters. III-Calibration with 2MASS

We present two absolute magnitude calibrations, $M_{J}$ and $M_{K_s}$, for red giants with the colour magnitude diagrams of five Galactic clusters with different metallicities i.e. M92, M13, M71, M67, and NGC 6791. The combination of the absolute magnitudes of the red giant sequences with the corresponding metallicities provides calibration for absolute magnitude estimation for red giants for a given colour. The calibrations for $M_{J}$ and $M_{K_s}$ are defined in the colour intervals $1.3\leq(V-J)_{0}\leq2.8$ and $1.75 \leq (V-K_{s})_{0}\leq 3.80$ mag, respectively, and they cover the metallicity interval $-2.15 \leq \lbrack Fe/H \rbrack \leq +0.37$ dex. The absolute magnitude residuals obtained by the application of the procedure to another set of Galactic clusters lie in the intervals $-0.08<\Delta M_{J}\leq +0.34$ and $-0.10< \Delta M_{K_s}\leq +0.27$ mag for $M_{J}$ and $M_{K_s}$, respectively. The means and standard deviations of the residuals are $= 0.137$ and $\sigma_{M_J}=0.080$, and $=0.109$ and $\sigma_{M_{K_{s}}}=0.123$ mag. The derived relations are applicable to stars older than 4 Gyr, the age of the youngest calibrating cluster.Comment: 20 pages, including 8 figures and 22 tables, accepted for publication in PASA. arXiv admin note: substantial text overlap with arXiv:1206.275

Schwarzschild models of the Sculptor dSph galaxy

We have developed a spherically symmetric dynamical model of a dwarf spheroidal galaxy using the Schwarzschild method. This type of modelling yields constraints both on the total mass distribution (e.g. enclosed mass and scale radius) as well as on the orbital structure of the system modelled (e.g. velocity anisotropy). Therefore not only can we derive the dark matter content of these systems, but also explore possible formation scenarios. Here we present preliminary results for the Sculptor dSph. We find that the mass of Sculptor within 1kpc is 8.5\times10^(7\pm0.05) M\odot, its anisotropy profile is tangentially biased and slightly more isotropic near the center. For an NFW profile, the preferred concentration (~15) is compatible with cosmological models. Very cuspy density profiles (steeper than NFW) are strongly disfavoured for Sculptor.Comment: 2 pages, 4 figures, to appear in the proceedings of "Assembling the Puzzle of the Milky Way", Le Grand Bornand (Apr. 17-22, 2011