Simulating the universe on an intercontinental grid of supercomputers

Understanding the universe is hampered by the elusiveness of its most common constituent, cold dark matter. Almost impossible to observe, dark matter can be studied effectively by means of simulation and there is probably no other research field where simulation has led to so much progress in the last decade. Cosmological N-body simulations are an essential tool for evolving density perturbations in the nonlinear regime. Simulating the formation of large-scale structures in the universe, however, is still a challenge due to the enormous dynamic range in spatial and temporal coordinates, and due to the enormous computer resources required. The dynamic range is generally dealt with by the hybridization of numerical techniques. We deal with the computational requirements by connecting two supercomputers via an optical network and make them operate as a single machine. This is challenging, if only for the fact that the supercomputers of our choice are separated by half the planet, as one is located in Amsterdam and the other is in Tokyo. The co-scheduling of the two computers and the 'gridification' of the code enables us to achieve a 90% efficiency for this distributed intercontinental supercomputer.Comment: Accepted for publication in IEEE Compute

Scientific Computing Meets Big Data Technology: An Astronomy Use Case

Scientific analyses commonly compose multiple single-process programs into a dataflow. An end-to-end dataflow of single-process programs is known as a many-task application. Typically, tools from the HPC software stack are used to parallelize these analyses. In this work, we investigate an alternate approach that uses Apache Spark -- a modern big data platform -- to parallelize many-task applications. We present Kira, a flexible and distributed astronomy image processing toolkit using Apache Spark. We then use the Kira toolkit to implement a Source Extractor application for astronomy images, called Kira SE. With Kira SE as the use case, we study the programming flexibility, dataflow richness, scheduling capacity and performance of Apache Spark running on the EC2 cloud. By exploiting data locality, Kira SE achieves a 2.5x speedup over an equivalent C program when analyzing a 1TB dataset using 512 cores on the Amazon EC2 cloud. Furthermore, we show that by leveraging software originally designed for big data infrastructure, Kira SE achieves competitive performance to the C implementation running on the NERSC Edison supercomputer. Our experience with Kira indicates that emerging Big Data platforms such as Apache Spark are a performant alternative for many-task scientific applications

The Zwicky Transient Facility: Surveys and Scheduler

We present a novel algorithm for scheduling the observations of time-domain imaging surveys. Our Integer Linear Programming approach optimizes an observing plan for an entire night by assigning targets to temporal blocks, enabling strict control of the number of exposures obtained per field and minimizing filter changes. A subsequent optimization step minimizes slew times between each observation. Our optimization metric self-consistently weights contributions from time-varying airmass, seeing, and sky brightness to maximize the transient discovery rate. We describe the implementation of this algorithm on the surveys of the Zwicky Transient Facility and present its on-sky performance.Comment: Published in PASP Focus Issue on the Zwicky Transient Facility (https://dx.doi.org/10.1088/1538-3873/ab0c2a). 13 Pages, 11 Figure

Rapid GRB Follow-up with the 2-m Robotic Liverpool Telescope

We present the capabilities of the 2-m robotic Liverpool Telescope (LT), owned and operated by Liverpool John Moores University and situated at ORM, La Palma. Robotic control and scheduling of the LT make it especially powerful for observations in time domain astrophysics including: (i) rapid response to Targets of Opportunity: Gamma Ray Bursts, novae, supernovae, comets; (ii) monitoring of variable objects on timescales from seconds to years, and (iii) observations simultaneous or coordinated with other facilities, both ground-based and from space. Following a GRB alert from the Gamma Ray Observatories HETE-2, INTEGRAL and Swift we implement a special over-ride mode which enables observations to commence in about a minute after the alert, including optical and near infrared imaging and spectroscopy. In particular, the combination of aperture, site, instrumentation and rapid response (aided by its rapid slew and fully-opening enclosure) makes the LT excellently suited to help solving the mystery of the origin of optically dark GRBs, for the investigation of short bursts (which currently do not have any confirmed optical counterparts) and for early optical spectroscopy of the GRB phenomenon in general. We briefly describe the LT's key position in the RoboNet-1.0 network of robotic telescopes.Comment: 6 pages, 2 figures, to appear in the proceedings of Interacting Binaries: Accretion, Evolution and Outcomes, 4-10 July 2004, Cefalu, Sicily, Italy, eds. Antonelli et a