Scientific Computing Meets Big Data Technology: An Astronomy Use Case

Scientific analyses commonly compose multiple single-process programs into a dataflow. An end-to-end dataflow of single-process programs is known as a many-task application. Typically, tools from the HPC software stack are used to parallelize these analyses. In this work, we investigate an alternate approach that uses Apache Spark -- a modern big data platform -- to parallelize many-task applications. We present Kira, a flexible and distributed astronomy image processing toolkit using Apache Spark. We then use the Kira toolkit to implement a Source Extractor application for astronomy images, called Kira SE. With Kira SE as the use case, we study the programming flexibility, dataflow richness, scheduling capacity and performance of Apache Spark running on the EC2 cloud. By exploiting data locality, Kira SE achieves a 2.5x speedup over an equivalent C program when analyzing a 1TB dataset using 512 cores on the Amazon EC2 cloud. Furthermore, we show that by leveraging software originally designed for big data infrastructure, Kira SE achieves competitive performance to the C implementation running on the NERSC Edison supercomputer. Our experience with Kira indicates that emerging Big Data platforms such as Apache Spark are a performant alternative for many-task scientific applications

The Multi-Epoch Nearby Cluster Survey: type Ia supernova rate measurement in z~0.1 clusters and the late-time delay time distribution

We describe the Multi-Epoch Nearby Cluster Survey (MENeaCS), designed to measure the cluster Type Ia supernova (SN Ia) rate in a sample of 57 X-ray selected galaxy clusters, with redshifts of 0.05 < z < 0.15. Utilizing our real time analysis pipeline, we spectroscopically confirmed twenty-three cluster SN Ia, four of which were intracluster events. Using our deep CFHT/Megacam imaging, we measured total stellar luminosities in each of our galaxy clusters, and we performed detailed supernova detection efficiency simulations. Bringing these ingredients together, we measure an overall cluster SN Ia rate within R_{200} (1 Mpc) of 0.042^{+0.012}_{-0.010}^{+0.010}_{-0.008} SNuM (0.049^{+0.016}_{-0.014}^{+0.005}_{-0.004} SNuM) and a SN Ia rate within red sequence galaxies of 0.041^{+0.015}_{-0.015}^{+0.005}_{-0.010} SNuM (0.041^{+0.019}_{-0.015}^{+0.005}_{-0.004} SNuM). The red sequence SN Ia rate is consistent with published rates in early type/elliptical galaxies in the `field'. Using our red sequence SN Ia rate, and other cluster SNe measurements in early type galaxies up to $z\sim1$, we derive the late time (>2 Gyr) delay time distribution (DTD) of SN Ia assuming a cluster early type galaxy star formation epoch of z_f=3. Assuming a power law form for the DTD, \Psi(t)\propto t^s, we find s=-1.62\pm0.54. This result is consistent with predictions for the double degenerate SN Ia progenitor scenario (s\sim-1), and is also in line with recent calculations for the double detonation explosion mechanism (s\sim-2). The most recent calculations of the single degenerate scenario delay time distribution predicts an order of magnitude drop off in SN Ia rate \sim 6-7 Gyr after stellar formation, and the observed cluster rates cannot rule this out.Comment: 35 pages, 14 figures, ApJ accepte

Intracluster supernovae in the Multi-epoch Nearby Cluster Survey

The Multi-Epoch Nearby Cluster Survey (MENeaCS) has discovered twenty-three cluster Type Ia supernovae (SNe) in the 58 X-ray selected galaxy clusters (0.05 < z < 0.15) surveyed. Four of our SN Ia events have no host galaxy on close inspection, and are likely intracluster SNe. Deep image stacks at the location of the candidate intracluster SNe put upper limits on the luminosities of faint hosts, with M_{r} > -13.0 mag and M_{g} > -12.5 mag in all cases. For such limits, the fraction of the cluster luminosity in faint dwarfs below our detection limit is <0.1%, assuming a standard cluster luminosity function. All four events occurred within ~600 kpc of the cluster center (projected), as defined by the position of the brightest cluster galaxy, and are more centrally concentrated than the cluster SN Ia population as a whole. After accounting for several observational biases that make intracluster SNe easier to discover and spectroscopically confirm, we calculate an intracluster stellar mass fraction of 0.16^{+0.13}_{-0.09} (68% CL) for all objects within R_{200}. If we assume that the intracluster stellar population is exclusively old, and the cluster galaxies themselves have a mix of stellar ages, we derive an upper limit on the intracluster stellar mass fraction of <0.47 (84% one-sided CL). When combined with the intragroup SNe results of McGee & Balogh, we confirm the declining intracluster stellar mass fraction as a function of halo mass reported by Gonzalez and collaborators. (Abridged)Comment: 24 pages, 8 figures, ApJ publishe

The Extreme Hosts of Extreme Supernovae

We use GALEX ultraviolet (UV) and optical integrated photometry of the hosts of seventeen luminous supernovae (LSNe, having peak M_V < -21) and compare them to a sample of 26,000 galaxies from a cross-match between the SDSS DR4 spectral catalog and GALEX interim release 1.1. We place the LSNe hosts on the galaxy NUV-r versus M_r color magnitude diagram (CMD) with the larger sample to illustrate how extreme they are. The LSN hosts appear to favor low-density regions of the galaxy CMD falling on the blue edge of the blue cloud toward the low luminosity end. From the UV-optical photometry, we estimate the star formation history of the LSN hosts. The hosts have moderately low star formation rates (SFRs) and low stellar masses (M_*) resulting in high specific star formation rates (sSFR). Compared with the larger sample, the LSN hosts occupy low-density regions of a diagram plotting sSFR versus M_* in the area having higher sSFR and lower M_*. This preference for low M_*, high sSFR hosts implies the LSNe are produced by an effect having to do with their local environment. The correlation of mass with metallicity suggests that perhaps wind-driven mass loss is the factor that prevents LSNe from arising in higher-mass, higher-metallicity hosts. The massive progenitors of the LSNe (>100 M_sun), by appearing in low-SFR hosts, are potential tests for theories of the initial mass function that limit the maximum mass of a star based on the SFR.Comment: 8 pages, 3 figures, 2 tables, accepted to ApJ, amended references and updated SN designation

The Sloan Digital Sky Survey-II Supernova Survey: Search Algorithm and Follow-up Observations

The Sloan Digital Sky Survey-II Supernova Survey has identified a large number of new transient sources in a 300 sq. deg. region along the celestial equator during its first two seasons of a three-season campaign. Multi-band (ugriz) light curves were measured for most of the sources, which include solar system objects, Galactic variable stars, active galactic nuclei, supernovae (SNe), and other astronomical transients. The imaging survey is augmented by an extensive spectroscopic follow-up program to identify SNe, measure their redshifts, and study the physical conditions of the explosions and their environment through spectroscopic diagnostics. During the survey, light curves are rapidly evaluated to provide an initial photometric type of the SNe, and a selected sample of sources are targeted for spectroscopic observations. In the first two seasons, 476 sources were selected for spectroscopic observations, of which 403 were identified as SNe. For the Type Ia SNe, the main driver for the Survey, our photometric typing and targeting efficiency is 90%. Only 6% of the photometric SN Ia candidates were spectroscopically classified as non-SN Ia instead, and the remaining 4% resulted in low signal-to-noise, unclassified spectra. This paper describes the search algorithm and the software, and the real-time processing of the SDSS imaging data. We also present the details of the supernova candidate selection procedures and strategies for follow-up spectroscopic and imaging observations of the discovered sources.Comment: Accepted for publication in The Astronomical Journal (66 pages, 13 figures); typos correcte

The hydrogen-poor superluminous supernova iPTF 13ajg and its host galaxy in absorption and emission

We present imaging and spectroscopy of a hydrogen-poor superluminous supernova (SLSN) discovered by the intermediate Palomar Transient Factory, iPTF 13ajg. At a redshift of z = 0.7403, derived from narrow absorption lines, iPTF 13ajg peaked at an absolute magnitude of M u, AB = -22.5, one of the most luminous supernovae to date. The observed bolometric peak luminosity of iPTF 13ajg is 3.2 × 1044 erg s-1, while the estimated total radiated energy is 1.3 × 1051 erg. We detect narrow absorption lines of Mg I, Mg II, and Fe II, associated with the cold interstellar medium in the host galaxy, at two different epochs with X-shooter at the Very Large Telescope. From Voigt profile fitting, we derive the column densities log N(Mg I) =11.94 ± 0.06, log N(Mg II) =14.7 ± 0.3, and log N(Fe II) =14.25 ± 0.10. These column densities, as well as the Mg I and Mg II equivalent widths of a sample of hydrogen-poor SLSNe taken from the literature, are at the low end of those derived for gamma-ray bursts (GRBs) whose progenitors are also thought to be massive stars. This suggests that the environments of hydrogen-poor SLSNe and GRBs are different. From the nondetection of Fe II fine-structure absorption lines, we derive a lower limit on the distance between the supernova and the narrow-line absorbing gas of 50 pc. The neutral gas responsible for the absorption in iPTF 13ajg exhibits a single narrow component with a low velocity width, ΔV = 76 km s-1, indicating a low-mass host galaxy. No host galaxy emission lines are detected, leading to an upper limit on the unobscured star formation rate (SFR) of SFR. Late-time imaging shows the iPTF 13ajg host galaxy to be faint, with g AB 27.0 and R AB ≥ 26.0 mag, corresponding to M B, Vega ≳ -17.7 mag. © 2014. The American Astronomical Society. All rights reserved.

Host galaxy identification for supernova surveys

Host galaxy identification is a crucial step for modern supernova (SN) surveys such as the Dark Energy Survey (DES) and the Large Synoptic Survey Telescope (LSST), which will discover SNe by the thousands. Spectroscopic resources are limited, so in the absence of real-time SN spectra these surveys must rely on host galaxy spectra to obtain accurate redshifts for the Hubble diagram and to improve photometric classification of SNe. In addition, SN luminosities are known to correlate with host-galaxy properties. Therefore, reliable identification of host galaxies is essential for cosmology and SN science. We simulate SN events and their locations within their host galaxies to develop and test methods for matching SNe to their hosts. We use both real and simulated galaxy catalog data from the Advanced Camera for Surveys General Catalog and MICECATv2.0, respectively. We also incorporate "hostless" SNe residing in undetected faint hosts into our analysis, with an assumed hostless rate of 5%. Our fully automated algorithm is run on catalog data and matches SNe to their hosts with 91% accuracy. We find that including a machine learning component, run after the initial matching algorithm, improves the accuracy (purity) of the matching to 97% with a 2% cost in efficiency (true positive rate). Although the exact results are dependent on the details of the survey and the galaxy catalogs used, the method of identifying host galaxies we outline here can be applied to any transient survey

Discovery of an Unusual Optical Transient with the Hubble Space Telescope

We present observations of SCP 06F6, an unusual optical transient discovered during the Hubble Space Telescope Cluster Supernova Survey. The transient brightened over a period of ~;;100 days, reached a peak magnitude of ~;;21.0 in both i_775 and z_850, and then declined over a similar timescale. There is no host galaxy or progenitor star detected at the location of the transient to a 3 sigma upper limit of i_775 = 26.4 and z_850 = 26.1, giving a corresponding lower limit on the flux increase of a factor of ~;;120. Multiple spectra show five broad absorption bands between 4100 AA and 6500 AA and a mostly featureless continuum longward of 6500 AA. The shape of the lightcurve is inconsistent with microlensing. The transient's spectrum, in addition to being inconsistent with all known supernova types, is not matched to any spectrum in the Sloan Digital Sky Survey (SDSS) database. We suggest that the transient may be one of a new class

The Discovery of a Gravitationally Lensed Supernova Ia at Redshift 2.22

We present the discovery and measurements of a gravitationally lensed supernova (SN) behind the galaxy cluster MOO J1014+0038. Based on multi-band Hubble Space Telescope and Very Large Telescope (VLT) photometry of the supernova, and VLT spectroscopy of the host galaxy, we find a 97.5% probability that this SN is a SN Ia, and a 2.5% chance of a CC SN. Our typing algorithm combines the shape and color of the light curve with the expected rates of each SN type in the host galaxy. With a redshift of 2.2216, this is the highest redshift SN Ia discovered with a spectroscopic host-galaxy redshift. A further distinguishing feature is that the lensing cluster, at redshift 1.23, is the most distant to date to have an amplified SN. The SN lies in the middle of the color and light-curve shape distributions found at lower redshift, disfavoring strong evolution to z = 2.22. We estimate an amplification due to gravitational lensing of 2.8+0.6-0.5 (1.10 +- 0.23 mag)---compatible with the value estimated from the weak-lensing-derived mass and the mass-concentration relation from LambdaCDM simulations---making it the most amplified SN Ia discovered behind a galaxy cluster