2,371 research outputs found

    Vaex: Big Data exploration in the era of Gaia

    Get PDF
    We present a new Python library called vaex, to handle extremely large tabular datasets, such as astronomical catalogues like the Gaia catalogue, N-body simulations or any other regular datasets which can be structured in rows and columns. Fast computations of statistics on regular N-dimensional grids allows analysis and visualization in the order of a billion rows per second. We use streaming algorithms, memory mapped files and a zero memory copy policy to allow exploration of datasets larger than memory, e.g. out-of-core algorithms. Vaex allows arbitrary (mathematical) transformations using normal Python expressions and (a subset of) numpy functions which are lazily evaluated and computed when needed in small chunks, which avoids wasting of RAM. Boolean expressions (which are also lazily evaluated) can be used to explore subsets of the data, which we call selections. Vaex uses a similar DataFrame API as Pandas, a very popular library, which helps migration from Pandas. Visualization is one of the key points of vaex, and is done using binned statistics in 1d (e.g. histogram), in 2d (e.g. 2d histograms with colormapping) and 3d (using volume rendering). Vaex is split in in several packages: vaex-core for the computational part, vaex-viz for visualization mostly based on matplotlib, vaex-jupyter for visualization in the Jupyter notebook/lab based in IPyWidgets, vaex-server for the (optional) client-server communication, vaex-ui for the Qt based interface, vaex-hdf5 for hdf5 based memory mapped storage, vaex-astro for astronomy related selections, transformations and memory mapped (column based) fits storage. Vaex is open source and available under MIT license on github, documentation and other information can be found on the main website: https://vaex.io, https://docs.vaex.io or https://github.com/maartenbreddels/vaexComment: 14 pages, 8 figures, Submitted to A&A, interactive version of Fig 4: https://vaex.io/paper/fig

    CacophonyViz: Visualisation of Birdsong Derived Ecological Health Indicators

    Get PDF
    The purpose of this work was to create an easy to interpret visualisation of a simple index that represents the quantity and quality of bird life in New Zealand. The index was calculated from an algorithm that assigned various weights to each species of bird. This work is important as it forms a part of the ongoing work by the Cacophony Project which aims to eradicate pests that currently destroy New Zealand native birds and their habitat. The map will be used to promote the Cacophony project to a wide public audience and encourage their participation by giving relevant feedback on the effects of intervention such as planting and trapping in their communities. The Design Science methodology guided this work through the creation of a series of prototypes that through their evaluation built on lessons learnt at each stage resulting in a final artifact that successfully displayed the index at various locations across a map of New Zealand. It is concluded that the artifact is ready and suitable for deployment once the availability of real data from the automatic analysis of audio recordings from multiple locations becomes available

    CacophonyViz : Visualisation of birdsong derived ecological health indicators

    Get PDF
    The purpose of this work was to create an easy to interpret visualisation of a simple index that represents the quantity and quality of bird life in New Zealand. The index was calculated from an algorithm that assigned various weights to each species of bird. This work is important as it forms a part of the ongoing work by the Cacophony Project which aims to eradicate pests that currently destroy New Zealand native birds and their habitat. The map will be used to promote the Cacophony project to a wide public audience and encourage their participation by giving relevant feedback on the effects of intervention such as planting and trapping in their communities. The Design Science methodology guided this work through the creation of a series of prototypes that through their evaluation built on lessons learnt at each stage resulting in a final artifact that successfully displayed the index at various locations across a map of New Zealand. It is concluded that the artifact is ready and suitable for deployment once the availability of real data from the automatic analysis of audio recordings from multiple locations becomes available

    Scalable visualisation methods for modern Generalized Additive Models

    Full text link
    In the last two decades the growth of computational resources has made it possible to handle Generalized Additive Models (GAMs) that formerly were too costly for serious applications. However, the growth in model complexity has not been matched by improved visualisations for model development and results presentation. Motivated by an industrial application in electricity load forecasting, we identify the areas where the lack of modern visualisation tools for GAMs is particularly severe, and we address the shortcomings of existing methods by proposing a set of visual tools that a) are fast enough for interactive use, b) exploit the additive structure of GAMs, c) scale to large data sets and d) can be used in conjunction with a wide range of response distributions. All the new visual methods proposed in this work are implemented by the mgcViz R package, which can be found on the Comprehensive R Archive Network

    VIOLA - A multi-purpose and web-based visualization tool for neuronal-network simulation output

    Full text link
    Neuronal network models and corresponding computer simulations are invaluable tools to aid the interpretation of the relationship between neuron properties, connectivity and measured activity in cortical tissue. Spatiotemporal patterns of activity propagating across the cortical surface as observed experimentally can for example be described by neuronal network models with layered geometry and distance-dependent connectivity. The interpretation of the resulting stream of multi-modal and multi-dimensional simulation data calls for integrating interactive visualization steps into existing simulation-analysis workflows. Here, we present a set of interactive visualization concepts called views for the visual analysis of activity data in topological network models, and a corresponding reference implementation VIOLA (VIsualization Of Layer Activity). The software is a lightweight, open-source, web-based and platform-independent application combining and adapting modern interactive visualization paradigms, such as coordinated multiple views, for massively parallel neurophysiological data. For a use-case demonstration we consider spiking activity data of a two-population, layered point-neuron network model subject to a spatially confined excitation originating from an external population. With the multiple coordinated views, an explorative and qualitative assessment of the spatiotemporal features of neuronal activity can be performed upfront of a detailed quantitative data analysis of specific aspects of the data. Furthermore, ongoing efforts including the European Human Brain Project aim at providing online user portals for integrated model development, simulation, analysis and provenance tracking, wherein interactive visual analysis tools are one component. Browser-compatible, web-technology based solutions are therefore required. Within this scope, with VIOLA we provide a first prototype.Comment: 38 pages, 10 figures, 3 table

    GeoZui3D: Data Fusion for Interpreting Oceanographic Data

    Get PDF
    GeoZui3D stands for Geographic Zooming User Interface. It is a new visualization software system designed for interpreting multiple sources of 3D data. The system supports gridded terrain models, triangular meshes, curtain plots, and a number of other display objects. A novel center of workspace interaction method unifies a number of aspects of the interface. It creates a simple viewpoint control method, it helps link multiple views, and is ideal for stereoscopic viewing. GeoZui3D has a number of features to support real-time input. Through a CORBA interface external entities can influence the position and state of objects in the display. Extra windows can be attached to moving objects allowing for their position and data to be monitored. We describe the application of this system for heterogeneous data fusion, for multibeam QC and for ROV/AUV monitoring

    Integrated spectra extraction based on signal-to-noise optimization using Integral Field Spectroscopy

    Full text link
    We propose and explore the potential of a method to extract high signal-to-noise (S/N) integrated spectra related to physical and/or morphological regions on a 2-dimensional field using Integral Field Spectroscopy (IFS) observations by employing an optimization procedure based on either continuum (stellar) or line (nebular) emission features. The optimization method is applied to a set of IFS VLT-VIMOS observations of (U)LIRG galaxies, describing the advantages of the optimization by comparing the results with a fixed-aperture, single spectrum case, and by implementing some statistical tests. We demonstrate that the S/N of the IFS optimized integrated spectra is significantly enhanced when compared with the single aperture unprocessed case. We provide an iterative user-friendly and versatile IDL algorithm that allows the user to spatially integrate spectra following more standard procedures. This is made available to the community as part of the PINGSoft IFS software package.Comment: Accepted for publication in Astronomy & Astrophysics, 12 pages, 7 figure
    • …
    corecore