432,562 research outputs found
Vaex: Big Data exploration in the era of Gaia
We present a new Python library called vaex, to handle extremely large
tabular datasets, such as astronomical catalogues like the Gaia catalogue,
N-body simulations or any other regular datasets which can be structured in
rows and columns. Fast computations of statistics on regular N-dimensional
grids allows analysis and visualization in the order of a billion rows per
second. We use streaming algorithms, memory mapped files and a zero memory copy
policy to allow exploration of datasets larger than memory, e.g. out-of-core
algorithms. Vaex allows arbitrary (mathematical) transformations using normal
Python expressions and (a subset of) numpy functions which are lazily evaluated
and computed when needed in small chunks, which avoids wasting of RAM. Boolean
expressions (which are also lazily evaluated) can be used to explore subsets of
the data, which we call selections. Vaex uses a similar DataFrame API as
Pandas, a very popular library, which helps migration from Pandas.
Visualization is one of the key points of vaex, and is done using binned
statistics in 1d (e.g. histogram), in 2d (e.g. 2d histograms with colormapping)
and 3d (using volume rendering). Vaex is split in in several packages:
vaex-core for the computational part, vaex-viz for visualization mostly based
on matplotlib, vaex-jupyter for visualization in the Jupyter notebook/lab based
in IPyWidgets, vaex-server for the (optional) client-server communication,
vaex-ui for the Qt based interface, vaex-hdf5 for hdf5 based memory mapped
storage, vaex-astro for astronomy related selections, transformations and
memory mapped (column based) fits storage. Vaex is open source and available
under MIT license on github, documentation and other information can be found
on the main website: https://vaex.io, https://docs.vaex.io or
https://github.com/maartenbreddels/vaexComment: 14 pages, 8 figures, Submitted to A&A, interactive version of Fig 4:
https://vaex.io/paper/fig
Recommended from our members
Self-interference fluorescence microscopy: three dimensional fluorescence imaging without depth scanning
We present a new method for high-resolution, three-dimensional fluorescence imaging. In contrast to beam-scanning confocal microscopy, where the laser focus must be scanned both laterally and axially to collect a volume, we obtain depth information without the necessity of depth scanning. In this method, the emitted fluorescence is collected in the backward direction and is sent through a phase plate that encodes the depth information into the phase of a spectrally resolved interference pattern. We demonstrate that decoding this phase information allows for depth localization accuracy better than 4 µm over a 500 µm depth-of-field. In a high numerical aperture configuration with a much smaller depth of field, a localization accuracy of tens of nanometers can be achieved. This approach is ideally suited for miniature endoscopes, where space limitations at the endoscope tip render depth scanning difficult. We illustrate the potential for 3D visualization of complex biological samples by constructing a three-dimensional volume of the microvasculature of ex vivo murine heart tissue from a single 2D scan
Self-interference fluorescence microscopy: three dimensional fluorescence imaging without depth scanning
We present a new method for high-resolution, three-dimensional fluorescence imaging. In contrast to beam-scanning confocal microscopy, where the laser focus must be scanned both laterally and axially to collect a volume, we obtain depth information without the necessity of depth scanning. In this method, the emitted fluorescence is collected in the backward direction and is sent through a phase plate that encodes the depth information into the phase of a spectrally resolved interference pattern. We demonstrate that decoding this phase information allows for depth localization accuracy better than 4 μm over a 500 μm depth-of-field. In a high numerical aperture configuration with a much smaller depth of field, a localization accuracy of tens of nanometers can be achieved. This approach is ideally suited for miniature endoscopes, where space limitations at the endoscope tip render depth scanning difficult. We illustrate the potential for 3D visualization of complex biological samples by constructing a threedimensional volume of the microvasculature of ex vivo murine heart tissue from a single 2D scan. © 2012 Optical Society of America
Detection of Microcalcifications in Digital Breast Tomosynthesis using Faster R-CNN and 3D Volume Rendering
Microcalcification clusters (MCs) are one of the most important biomarkers for breast cancer and Digital
Breast Tomosynthesis (DBT) has consolidated its role in breast cancer imaging. As there are mixed
observations about MCs detection using DBT, it is important to develop tools that improve this task.
Furthermore, the visualization mode of MCs is also crucial, as their diagnosis is associated with their 3D
morphology. In this work, DBT data from a public database were used to train a faster region-based
convolutional neural network (R-CNN) to locate MCs in entire DBT. Additionally, the detected MCs were
further analyzed through standard 2D visualization and 3D volume rendering (VR) specifically developed for
DBT data. For MCs detection, the sensitivity of our Faster R-CNN was 60% with 4 false positives. These
preliminary results are very promising and can be further improved. On the other hand, the 3D VR
visualization provided important information, with higher quality and discernment of the detected MCs. The
developed pipeline may help radiologists since (1) it indicates specific breast regions with possible lesions
that deserve additional attention and (2) as the rendering of the MCs is similar to a segmentation, a detailed
complementary analysis of their 3D morphology is possible
Synergistic Visualization And Quantitative Analysis Of Volumetric Medical Images
The medical diagnosis process starts with an interview with the patient, and continues with the physical exam. In practice, the medical professional may require additional screenings to precisely diagnose. Medical imaging is one of the most frequently used non-invasive screening methods to acquire insight of human body. Medical imaging is not only essential for accurate diagnosis, but also it can enable early prevention. Medical data visualization refers to projecting the medical data into a human understandable format at mediums such as 2D or head-mounted displays without causing any interpretation which may lead to clinical intervention. In contrast to the medical visualization, quantification refers to extracting the information in the medical scan to enable the clinicians to make fast and accurate decisions. Despite the extraordinary process both in medical visualization and quantitative radiology, efforts to improve these two complementary fields are often performed independently and synergistic combination is under-studied. Existing image-based software platforms mostly fail to be used in routine clinics due to lack of a unified strategy that guides clinicians both visually and quan- titatively. Hence, there is an urgent need for a bridge connecting the medical visualization and automatic quantification algorithms in the same software platform. In this thesis, we aim to fill this research gap by visualizing medical images interactively from anywhere, and performing a fast, accurate and fully-automatic quantification of the medical imaging data. To end this, we propose several innovative and novel methods. Specifically, we solve the following sub-problems of the ul- timate goal: (1) direct web-based out-of-core volume rendering, (2) robust, accurate, and efficient learning based algorithms to segment highly pathological medical data, (3) automatic landmark- ing for aiding diagnosis and surgical planning and (4) novel artificial intelligence algorithms to determine the sufficient and necessary data to derive large-scale problems
Organizing gene literature retrieval, profiling, and visualization training workshops for early career researchers
Developing the skills needed to effectively search and extract information from biomedical literature is essential for early-career researchers. It is, for instance, on this basis that the novelty of experimental results, and therefore publishing opportunities, can be evaluated. Given the unprecedented volume of publications in the field of biomedical research, new systematic approaches need to be devised and adopted for the retrieval and curation of literature relevant to a specific theme. Here we describe a hands-on training curriculum aimed at retrieval, profiling, and visualization of literature associated with a given topic. This curriculum was implemented in a workshop in January 2021. We provide supporting material and step-by-step implementation guidelines with the ISG15 gene literature serving as an illustrative use case. Through participation in such a workshop, trainees can learn: 1) to build and troubleshoot PubMed queries in order to retrieve the literature associated with a gene of interest; 2) to identify key concepts relevant to given themes (such as cell types, diseases, and biological processes); 3) to measure the prevalence of these concepts in the gene literature; 4) to extract key information from relevant articles, and 5) to develop a background section or summary on the basis of this information. Finally, trainees can learn to consolidate the structured information captured through this process for presentation via an interactive web application
- …