2,371 research outputs found
Vaex: Big Data exploration in the era of Gaia
We present a new Python library called vaex, to handle extremely large
tabular datasets, such as astronomical catalogues like the Gaia catalogue,
N-body simulations or any other regular datasets which can be structured in
rows and columns. Fast computations of statistics on regular N-dimensional
grids allows analysis and visualization in the order of a billion rows per
second. We use streaming algorithms, memory mapped files and a zero memory copy
policy to allow exploration of datasets larger than memory, e.g. out-of-core
algorithms. Vaex allows arbitrary (mathematical) transformations using normal
Python expressions and (a subset of) numpy functions which are lazily evaluated
and computed when needed in small chunks, which avoids wasting of RAM. Boolean
expressions (which are also lazily evaluated) can be used to explore subsets of
the data, which we call selections. Vaex uses a similar DataFrame API as
Pandas, a very popular library, which helps migration from Pandas.
Visualization is one of the key points of vaex, and is done using binned
statistics in 1d (e.g. histogram), in 2d (e.g. 2d histograms with colormapping)
and 3d (using volume rendering). Vaex is split in in several packages:
vaex-core for the computational part, vaex-viz for visualization mostly based
on matplotlib, vaex-jupyter for visualization in the Jupyter notebook/lab based
in IPyWidgets, vaex-server for the (optional) client-server communication,
vaex-ui for the Qt based interface, vaex-hdf5 for hdf5 based memory mapped
storage, vaex-astro for astronomy related selections, transformations and
memory mapped (column based) fits storage. Vaex is open source and available
under MIT license on github, documentation and other information can be found
on the main website: https://vaex.io, https://docs.vaex.io or
https://github.com/maartenbreddels/vaexComment: 14 pages, 8 figures, Submitted to A&A, interactive version of Fig 4:
https://vaex.io/paper/fig
CacophonyViz: Visualisation of Birdsong Derived Ecological Health Indicators
The purpose of this work was to create an easy to interpret visualisation of a simple index that represents the quantity and quality of bird life in New Zealand. The index was calculated from an algorithm that assigned various weights to each species of bird.
This work is important as it forms a part of the ongoing work by the Cacophony Project which aims to eradicate pests that currently destroy New Zealand native birds and their habitat. The map will be used to promote the Cacophony project to a wide public audience and encourage their participation by giving relevant feedback on the effects of intervention such as planting and trapping in their communities.
The Design Science methodology guided this work through the creation of a series of prototypes that through their evaluation built on lessons learnt at each stage resulting in a final artifact that successfully displayed the index at various locations across a map of New Zealand.
It is concluded that the artifact is ready and suitable for deployment once the availability of real data from the automatic analysis of audio recordings from multiple locations becomes available
CacophonyViz : Visualisation of birdsong derived ecological health indicators
The purpose of this work was to create an easy to interpret visualisation of a simple index that represents the quantity and quality of bird life in New Zealand. The index
was calculated from an algorithm that assigned various weights to each species of
bird.
This work is important as it forms a part of the ongoing work by the Cacophony Project which aims to eradicate pests that currently destroy New Zealand native birds and their habitat. The map will be used to promote the Cacophony project to a wide public audience and encourage their participation by giving relevant feedback on the
effects of intervention such as planting and trapping in their communities.
The Design Science methodology guided this work through the creation of a series of prototypes that through their evaluation built on lessons learnt at each stage resulting
in a final artifact that successfully displayed the index at various locations across a map of New Zealand.
It is concluded that the artifact is ready and suitable for deployment once the availability of real data from the automatic analysis of audio recordings from multiple
locations becomes available
Scalable visualisation methods for modern Generalized Additive Models
In the last two decades the growth of computational resources has made it
possible to handle Generalized Additive Models (GAMs) that formerly were too
costly for serious applications. However, the growth in model complexity has
not been matched by improved visualisations for model development and results
presentation. Motivated by an industrial application in electricity load
forecasting, we identify the areas where the lack of modern visualisation tools
for GAMs is particularly severe, and we address the shortcomings of existing
methods by proposing a set of visual tools that a) are fast enough for
interactive use, b) exploit the additive structure of GAMs, c) scale to large
data sets and d) can be used in conjunction with a wide range of response
distributions. All the new visual methods proposed in this work are implemented
by the mgcViz R package, which can be found on the Comprehensive R Archive
Network
VIOLA - A multi-purpose and web-based visualization tool for neuronal-network simulation output
Neuronal network models and corresponding computer simulations are invaluable
tools to aid the interpretation of the relationship between neuron properties,
connectivity and measured activity in cortical tissue. Spatiotemporal patterns
of activity propagating across the cortical surface as observed experimentally
can for example be described by neuronal network models with layered geometry
and distance-dependent connectivity. The interpretation of the resulting stream
of multi-modal and multi-dimensional simulation data calls for integrating
interactive visualization steps into existing simulation-analysis workflows.
Here, we present a set of interactive visualization concepts called views for
the visual analysis of activity data in topological network models, and a
corresponding reference implementation VIOLA (VIsualization Of Layer Activity).
The software is a lightweight, open-source, web-based and platform-independent
application combining and adapting modern interactive visualization paradigms,
such as coordinated multiple views, for massively parallel neurophysiological
data. For a use-case demonstration we consider spiking activity data of a
two-population, layered point-neuron network model subject to a spatially
confined excitation originating from an external population. With the multiple
coordinated views, an explorative and qualitative assessment of the
spatiotemporal features of neuronal activity can be performed upfront of a
detailed quantitative data analysis of specific aspects of the data.
Furthermore, ongoing efforts including the European Human Brain Project aim at
providing online user portals for integrated model development, simulation,
analysis and provenance tracking, wherein interactive visual analysis tools are
one component. Browser-compatible, web-technology based solutions are therefore
required. Within this scope, with VIOLA we provide a first prototype.Comment: 38 pages, 10 figures, 3 table
GeoZui3D: Data Fusion for Interpreting Oceanographic Data
GeoZui3D stands for Geographic Zooming User Interface. It is a new visualization software system designed for interpreting multiple sources of 3D data. The system supports gridded terrain models, triangular meshes, curtain plots, and a number of other display objects. A novel center of workspace interaction method unifies a number of aspects of the interface. It creates a simple viewpoint control method, it helps link multiple views, and is ideal for stereoscopic viewing. GeoZui3D has a number of features to support real-time input. Through a CORBA interface external entities can influence the position and state of objects in the display. Extra windows can be attached to moving objects allowing for their position and data to be monitored. We describe the application of this system for heterogeneous data fusion, for multibeam QC and for ROV/AUV monitoring
Integrated spectra extraction based on signal-to-noise optimization using Integral Field Spectroscopy
We propose and explore the potential of a method to extract high
signal-to-noise (S/N) integrated spectra related to physical and/or
morphological regions on a 2-dimensional field using Integral Field
Spectroscopy (IFS) observations by employing an optimization procedure based on
either continuum (stellar) or line (nebular) emission features. The
optimization method is applied to a set of IFS VLT-VIMOS observations of
(U)LIRG galaxies, describing the advantages of the optimization by comparing
the results with a fixed-aperture, single spectrum case, and by implementing
some statistical tests. We demonstrate that the S/N of the IFS optimized
integrated spectra is significantly enhanced when compared with the single
aperture unprocessed case. We provide an iterative user-friendly and versatile
IDL algorithm that allows the user to spatially integrate spectra following
more standard procedures. This is made available to the community as part of
the PINGSoft IFS software package.Comment: Accepted for publication in Astronomy & Astrophysics, 12 pages, 7
figure
- …