3,245 research outputs found
Data Access for LIGO on the OSG
During 2015 and 2016, the Laser Interferometer Gravitational-Wave Observatory
(LIGO) conducted a three-month observing campaign. These observations delivered
the first direct detection of gravitational waves from binary black hole
mergers. To search for these signals, the LIGO Scientific Collaboration uses
the PyCBC search pipeline. To deliver science results in a timely manner, LIGO
collaborated with the Open Science Grid (OSG) to distribute the required
computation across a series of dedicated, opportunistic, and allocated
resources. To deliver the petabytes necessary for such a large-scale
computation, our team deployed a distributed data access infrastructure based
on the XRootD server suite and the CernVM File System (CVMFS). This data access
strategy grew from simply accessing remote storage to a POSIX-based interface
underpinned by distributed, secure caches across the OSG.Comment: 6 pages, 3 figures, submitted to PEARC1
Building Near-Real-Time Processing Pipelines with the Spark-MPI Platform
Advances in detectors and computational technologies provide new
opportunities for applied research and the fundamental sciences. Concurrently,
dramatic increases in the three Vs (Volume, Velocity, and Variety) of
experimental data and the scale of computational tasks produced the demand for
new real-time processing systems at experimental facilities. Recently, this
demand was addressed by the Spark-MPI approach connecting the Spark
data-intensive platform with the MPI high-performance framework. In contrast
with existing data management and analytics systems, Spark introduced a new
middleware based on resilient distributed datasets (RDDs), which decoupled
various data sources from high-level processing algorithms. The RDD middleware
significantly advanced the scope of data-intensive applications, spreading from
SQL queries to machine learning to graph processing. Spark-MPI further extended
the Spark ecosystem with the MPI applications using the Process Management
Interface. The paper explores this integrated platform within the context of
online ptychographic and tomographic reconstruction pipelines.Comment: New York Scientific Data Summit, August 6-9, 201
Analysis and implementation of the Large Scale Video-on-Demand System
Next Generation Network (NGN) provides multimedia services over broadband
based networks, which supports high definition TV (HDTV), and DVD quality
video-on-demand content. The video services are thus seen as merging mainly
three areas such as computing, communication, and broadcasting. It has numerous
advantages and more exploration for the large-scale deployment of
video-on-demand system is still needed. This is due to its economic and design
constraints. It's need significant initial investments for full service
provision. This paper presents different estimation for the different
topologies and it require efficient planning for a VOD system network. The
methodology investigates the network bandwidth requirements of a VOD system
based on centralized servers, and distributed local proxies. Network traffic
models are developed to evaluate the VOD system's operational bandwidth
requirements for these two network architectures. This paper present an
efficient estimation of the of the bandwidth requirement for the different
architectures.Comment: 9 pages, 8 figure
- …