3,053 research outputs found
Building Near-Real-Time Processing Pipelines with the Spark-MPI Platform
Advances in detectors and computational technologies provide new
opportunities for applied research and the fundamental sciences. Concurrently,
dramatic increases in the three Vs (Volume, Velocity, and Variety) of
experimental data and the scale of computational tasks produced the demand for
new real-time processing systems at experimental facilities. Recently, this
demand was addressed by the Spark-MPI approach connecting the Spark
data-intensive platform with the MPI high-performance framework. In contrast
with existing data management and analytics systems, Spark introduced a new
middleware based on resilient distributed datasets (RDDs), which decoupled
various data sources from high-level processing algorithms. The RDD middleware
significantly advanced the scope of data-intensive applications, spreading from
SQL queries to machine learning to graph processing. Spark-MPI further extended
the Spark ecosystem with the MPI applications using the Process Management
Interface. The paper explores this integrated platform within the context of
online ptychographic and tomographic reconstruction pipelines.Comment: New York Scientific Data Summit, August 6-9, 201
High Performance Algorithms for Counting Collisions and Pairwise Interactions
The problem of counting collisions or interactions is common in areas as
computer graphics and scientific simulations. Since it is a major bottleneck in
applications of these areas, a lot of research has been carried out on such
subject, mainly focused on techniques that allow calculations to be performed
within pruned sets of objects. This paper focuses on how interaction
calculation (such as collisions) within these sets can be done more efficiently
than existing approaches. Two algorithms are proposed: a sequential algorithm
that has linear complexity at the cost of high memory usage; and a parallel
algorithm, mathematically proved to be correct, that manages to use GPU
resources more efficiently than existing approaches. The proposed and existing
algorithms were implemented, and experiments show a speedup of 21.7 for the
sequential algorithm (on small problem size), and 1.12 for the parallel
proposal (large problem size). By improving interaction calculation, this work
contributes to research areas that promote interconnection in the modern world,
such as computer graphics and robotics.Comment: Accepted in ICCS 2019 and published in Springer's LNCS series.
Supplementary content at https://mjsaldanha.com/articles/1-hpc-ssp
German Science Center for the Solar Dynamics Observatory
A data and computation center for helioseismology has been set up at the Max
Planck Institute for Solar System Research in Germany to prepare for the SDO
mission. Here we present the system infrastructure and the scientific aims of
this project, which is funded through grants from the German Aerospace Center
and the European Research Council
An overview of the planned CCAT software system
CCAT will be a 25m diameter sub-millimeter telescope capable of operating in
the 0.2 to 2.1mm wavelength range. It will be located at an altitude of 5600m
on Cerro Chajnantor in northern Chile near the ALMA site. The anticipated first
generation instruments include large format (60,000 pixel) kinetic inductance
detector (KID) cameras, a large format heterodyne array and a direct detection
multi-object spectrometer. The paper describes the architecture of the CCAT
software and the development strategy.Comment: 17 pages, 6 figures, to appear in Software and Cyberinfrastructure
for Astronomy III, Chiozzi & Radziwill (eds), Proc. SPIE 9152, paper ID
9152-10
- …