53 research outputs found
Inter-cluster filaments in a CDM Universe
The large--scale structure (LSS) in the Universe comprises a complicated
filamentary network of matter. We study this network using a high--resolution
simulation of structure formation of a Cold Dark Matter cosmology. We
investigate the distribution of matter between neighbouring large haloes whose
masses are comparable to massive clusters of galaxies. We identify a total of
228 filaments between neighbouring clusters. Roughly half of the filaments are
either warped or lie off the cluster--cluster axis. We find that straight
filaments on the average are shorter than warped ones. More massive clusters
are connected to more filaments than less massive ones on average. This finding
indicates that the most massive clusters form at the intersections of the
filamentary backbone of LSS. For straight filaments, we compute mass profiles.
Radial profiles show a fairly well--defined radius, , beyond which the
profiles follow an power law fairly closely. For the majority of
filaments, lies between 1.5 Mpc and 2.0 Mpc. The
enclosed overdensity inside varies between a few times up to 25 times
mean density, independent of the length of the filaments. Along the filaments'
axes, material is not distributed uniformly. Towards the clusters, the density
rises, indicating the presence of the cluster infall regions. In addition, we
also find some sheet--like connections between clusters. In roughly a fifth of
all cluster--cluster connections where we could not identify a filament or
sheet, projection effects lead to filamentary structures in the projected mass
distribution. (abridged)Comment: 10 pages, 18 figures; submitted to MNRAS; updated: final version,
accepted for publicatio
Astronomy in the Cloud: Using MapReduce for Image Coaddition
In the coming decade, astronomical surveys of the sky will generate tens of
terabytes of images and detect hundreds of millions of sources every night. The
study of these sources will involve computation challenges such as anomaly
detection and classification, and moving object tracking. Since such studies
benefit from the highest quality data, methods such as image coaddition
(stacking) will be a critical preprocessing step prior to scientific
investigation. With a requirement that these images be analyzed on a nightly
basis to identify moving sources or transient objects, these data streams
present many computational challenges. Given the quantity of data involved, the
computational load of these problems can only be addressed by distributing the
workload over a large number of nodes. However, the high data throughput
demanded by these applications may present scalability challenges for certain
storage architectures. One scalable data-processing method that has emerged in
recent years is MapReduce, and in this paper we focus on its popular
open-source implementation called Hadoop. In the Hadoop framework, the data is
partitioned among storage attached directly to worker nodes, and the processing
workload is scheduled in parallel on the nodes that contain the required input
data. A further motivation for using Hadoop is that it allows us to exploit
cloud computing resources, e.g., Amazon's EC2. We report on our experience
implementing a scalable image-processing pipeline for the SDSS imaging database
using Hadoop. This multi-terabyte imaging dataset provides a good testbed for
algorithm development since its scope and structure approximate future surveys.
First, we describe MapReduce and how we adapted image coaddition to the
MapReduce framework. Then we describe a number of optimizations to our basic
approach and report experimental results comparing their performance.Comment: 31 pages, 11 figures, 2 table
Simulated LSST Survey of RR Lyrae Stars throughout the Local Group
We report on a study to determine the efficiency of the Large Synoptic Survey Telescope (LSST) to recover the periods, brightnesses, and shapes of RR Lyrae stars' light curves in the volume extending to heliocentric distances of 1.5 Mpc. We place the smoothed light curves of 30 type ab and 10 type c RR Lyrae stars in 1007 fields across the sky, each of which represents a different realization of the LSST sampling cadences, and that sample five particular observing modes. A light curve simulation tool was used to sample the idealized RR Lyrae stars' light curves, returning each as it would have been observed by LSST, including realistic photometric scatter, limiting magnitudes, and telescope downtime. We report here the period, brightness, and light curve shape recovery as a function of apparent magnitude and for survey lengths varying from 1 to 10 years. We find that 10 years of LSST data are sufficient to recover the pulsation periods with a fractional precision of ~10^(–5) for ≥90% of ab stars within ≈360 kpc of the Sun in Universal Cadence fields and out to ≈760 kpc for Deep Drilling fields. The 50% completeness level extends to ≈600 kpc and ≈1.0 Mpc for the same fields, respectively. For virtually all stars that had their periods recovered, their light curve shape parameter φ_31 was recovered with sufficient precision to also recover photometric metallicities to within 0.14 dex (the systematic error in the photometric relations). With RR Lyrae stars' periods and metallicities well measured to these distances, LSST will be able to search for halo streams and dwarf satellite galaxies over half of the Local Group, informing galaxy formation models and providing essential data for mapping the Galactic potential. This study also informs the LSST science operations plan for optimizing observing strategies to achieve particular science goals. We additionally present a new [Fe/H]-φ_31 photometric relation in the r band and a new and generally useful metric for defining period recovery for time domain surveys
Improving the LSST dithering pattern and cadence for dark energy studies
The Large Synoptic Survey Telescope (LSST) will explore the entire southern
sky over 10 years starting in 2022 with unprecedented depth and time sampling
in six filters, . Artificial power on the scale of the 3.5 deg LSST
field-of-view will contaminate measurements of baryonic acoustic oscillations
(BAO), which fall at the same angular scale at redshift . Using the
HEALPix framework, we demonstrate the impact of an "un-dithered" survey, in
which of each LSST field-of-view is overlapped by neighboring
observations, generating a honeycomb pattern of strongly varying survey depth
and significant artificial power on BAO angular scales. We find that adopting
large dithers (i.e., telescope pointing offsets) of amplitude close to the LSST
field-of-view radius reduces artificial structure in the galaxy distribution by
a factor of 10. We propose an observing strategy utilizing large dithers
within the main survey and minimal dithers for the LSST Deep Drilling Fields.
We show that applying various magnitude cutoffs can further increase survey
uniformity. We find that a magnitude cut of removes significant
spurious power from the angular power spectrum with a minimal reduction in the
total number of observed galaxies over the ten-year LSST run. We also determine
the effectiveness of the observing strategy for Type Ia SNe and predict that
the main survey will contribute 100,000 Type Ia SNe. We propose a
concentrated survey where LSST observes one-third of its main survey area each
year, increasing the number of main survey Type Ia SNe by a factor of
1.5, while still enabling the successful pursuit of other science
drivers.Comment: 9 pages, 6 figures, published in SPIE proceedings; corrected typo in
equation
Agile software development in an earned value world: a survival guide
Agile methodologies are current best practice in software development. They are favored for, among other reasons, preventing premature optimization by taking a somewhat short-term focus, and allowing frequent replans/reprioritizations of upcoming development work based on recent results and current backlog. At the same time, funding agencies prescribe earned value management accounting for large projects which, these days, inevitably include substantial software components. Earned Value approaches emphasize a more comprehensive and typically longer-range plan, and tend to characterize frequent replans and reprioritizations as indicative of problems. Here we describe the planning, execution and reporting framework used by the LSST Data Management team, that navigates these opposite tensions
Investigating interoperability of the LSST Data Management software stack with Astropy
The Large Synoptic Survey Telescope (LSST) will be an 8.4m optical survey telescope sited in Chile and capable of imaging the entire sky twice a week. The data rate of approximately 15TB per night and the requirements to both issue alerts on transient sources within 60 seconds of observing and create annual data releases means that automated data management systems and data processing pipelines are a key deliverable of the LSST construction project. The LSST data management software has been in development since 2004 and is based on a C++ core with a Python control layer. The software consists of nearly a quarter of a million lines of code covering the system from fundamental WCS and table libraries to pipeline environments and distributed process execution. The Astropy project began in 2011 as an attempt to bring together disparate open source Python projects and build a core standard infrastructure that can be used and built upon by the astronomy community. This project has been phenomenally successful in the years since it has begun and has grown to be the de facto standard for Python software in astronomy. Astropy brings with it considerable expectations from the community on how astronomy Python software should be developed and it is clear that by the time LSST is fully operational in the 2020s many of the prospective users of the LSST software stack will expect it to be fully interoperable with Astropy. In this paper we describe the overlap between the LSST science pipeline software and Astropy software and investigate areas where the LSST software provides new functionality. We also discuss the possibilities of re-engineering the LSST science pipeline software to build upon Astropy, including the option of contributing affliated packages
The Zwicky Transient Facility Alert Distribution System
The Zwicky Transient Facility (ZTF) survey generates real-time alerts for
optical transients, variables, and moving objects discovered in its wide-field
survey. We describe the ZTF alert stream distribution and processing
(filtering) system. The system uses existing open-source technologies developed
in industry: Kafka, a real-time streaming platform, and Avro, a binary
serialization format. The technologies used in this system provide a number of
advantages for the ZTF use case, including (1) built-in replication,
scalability, and stream rewind for the distribution mechanism; (2) structured
messages with strictly enforced schemas and dynamic typing for fast parsing;
and (3) a Python-based stream processing interface that is similar to batch for
a familiar and user-friendly plug-in filter system, all in a modular, primarily
containerized system. The production deployment has successfully supported
streaming up to 1.2 million alerts or roughly 70 GB of data per night, with
each alert available to a consumer within about 10 s of alert candidate
production. Data transfer rates of about 80,000 alerts/minute have been
observed. In this paper, we discuss this alert distribution and processing
system, the design motivations for the technology choices for the framework,
performance in production, and how this system may be generally suitable for
other alert stream use cases, including the upcoming Large Synoptic Survey
Telescope.Comment: Published in PASP Focus Issue on the Zwicky Transient Facility (doi:
10.1088/1538-3873/aae904). 9 Pages, 2 Figure
Software Architecture and System Design of Rubin Observatory
Starting from a description of the Rubin Observatory Data Management System
Architecture, and drawing on our experience with and involvement in a range of
other projects including Gaia, SDSS, UKIRT, and JCMT, we derive a series of
generic design patterns and lessons learned.Comment: 10 pages ADASS XXXII submissio
- …