Cyber-infrastructure to Support Science and Data Management for the Dark Energy Survey

The Dark Energy Survey (DES; operations 2009-2015) will address the nature of dark energy using four independent and complementary techniques: (1) a galaxy cluster survey over 4000 deg2 in collaboration with the South Pole Telescope Sunyaev-Zel'dovich effect mapping experiment, (2) a cosmic shear measurement over 5000 deg2, (3) a galaxy angular clustering measurement within redshift shells to redshift=1.35, and (4) distance measurements to 1900 supernovae Ia. The DES will produce 200 TB of raw data in four bands, These data will be processed into science ready images and catalogs and co-added into deeper, higher quality images and catalogs. In total, the DES dataset will exceed 1 PB, including a 100 TB catalog database that will serve as a key science analysis tool for the astronomy/cosmology community. The data rate, volume, and duration of the survey require a new type of data management (DM) system that (1) offers a high degree of automation and robustness and (2) leverages the existing high performance computing infrastructure to meet the project's DM targets. The DES DM system consists of (1) a grid-enabled, flexible and scalable middleware developed at NCSA for the broader scientific community, (2) astronomy modules that build upon community software, and (3) a DES archive to support automated processing and to serve DES catalogs and images to the collaboration and the public. In the recent DES Data Challenge 1 we deployed and tested the first version of the DES DM system, successfully reducing 700 GB of raw simulated images into 5 TB of reduced data products and cataloguing 50 million objects with calibrated astrometry and photometry.Comment: 12 pages, 3 color figures, 1 table. Published in SPIE vol. 627

Indiana University Pervasive Technology Institute – Research Technologies: XSEDE Service Provider and XSEDE subcontract report (PY1: 1 July 2011 to 30 June 2012)

Any opinions, findings, and conclusions or recommendations expressed in this material are those of the author(s) and do not necessarily reflect the views of the NSF or XSEDE leadership.This document is a summary of the activities of the Research Technologies division of UITS, a Service & Cyberinfrastructure Center affiliated with the Indiana University Pervasive Technology Institute, as part of the eXtreme Science and Engineering Discovery Environment (XSEDE) during XSEDE Program Year 1 (1 July 2011 – 30 June 2012). This document consists of three parts: - Section 2 of this document describes IU’s activities as an XSEDE Service Provider, using the format prescribed by XSEDE for reporting such activities. - Section 3 of this document describes IU’s activities as part of XSEDE management, operations, and support activities funded under a subcontract from the National Center for Supercomputer Applications (NCSA), the lead organization for XSEDE. This section is organized by the XSEDE Work Breakdown Structure (WBS) plan. - Appendix 1 is a summary table of IU’s education, outreach, and training events funded and supported in whole or in part by IU’s subcontract from NCSA as part of XSEDE.This document was developed with support from National Science Foundation (NSF) grant OCI-1053575

Identifying the time profile of everyday activities in the home using smart meter data

Activities are a descriptive term for the common ways households spend their time. Examples include cooking, doing laundry, or socialising. Smart meter data can be used to generate time profiles of activities that are meaningful to households’ own lived experience. Activities are therefore a lens through which energy feedback to households can be made salient and understandable. This paper demonstrates a multi-step methodology for inferring hourly time profiles of ten household activities using smart meter data, supplemented by individual appliance plug monitors and environmental sensors. First, household interviews, video ethnography, and technology surveys are used to identify appliances and devices in the home, and their roles in specific activities. Second, ‘ontologies’ are developed to map out the relationships between activities and technologies in the home. One or more technologies may indicate the occurrence of certain activities. Third, data from smart meters, plug monitors and sensor data are collected. Smart meter data measuring aggregate electricity use are disaggregated and processed together with the plug monitor and sensor data to identify when and for how long different activities are occurring. Sensor data are particularly useful for activities that are not always associated with an energy-using device. Fourth, the ontologies are applied to the disaggregated data to make inferences on hourly time profiles of ten everyday activities. These include washing, doing laundry, watching TV (reliably inferred), and cleaning, socialising, working (inferred with uncertainties). Fifth, activity time diaries and structured interviews are used to validate both the ontologies and the inferred activity time profiles. Two case study homes are used to illustrate the methodology using data collected as part of a UK trial of smart home technologies. The methodology is demonstrated to produce reliable time profiles of a range of domestic activities that are meaningful to households. The methodology also emphasises the value of integrating coded interview and video ethnography data into both the development of the activity inference process

Survey and Analysis of Production Distributed Computing Infrastructures

This report has two objectives. First, we describe a set of the production distributed infrastructures currently available, so that the reader has a basic understanding of them. This includes explaining why each infrastructure was created and made available and how it has succeeded and failed. The set is not complete, but we believe it is representative. Second, we describe the infrastructures in terms of their use, which is a combination of how they were designed to be used and how users have found ways to use them. Applications are often designed and created with specific infrastructures in mind, with both an appreciation of the existing capabilities provided by those infrastructures and an anticipation of their future capabilities. Here, the infrastructures we discuss were often designed and created with specific applications in mind, or at least specific types of applications. The reader should understand how the interplay between the infrastructure providers and the users leads to such usages, which we call usage modalities. These usage modalities are really abstractions that exist between the infrastructures and the applications; they influence the infrastructures by representing the applications, and they influence the ap- plications by representing the infrastructures

Towards a Benchmark for Fog Data Processing

Fog data processing systems provide key abstractions to manage data and event processing in the geo-distributed and heterogeneous fog environment. The lack of standardized benchmarks for such systems, however, hinders their development and deployment, as different approaches cannot be compared quantitatively. Existing cloud data benchmarks are inadequate for fog computing, as their focus on workload specification ignores the tight integration of application and infrastructure inherent in fog computing. In this paper, we outline an approach to a fog-native data processing benchmark that combines workload specifications with infrastructure specifications. This holistic approach allows researchers and engineers to quantify how a software approach performs for a given workload on given infrastructure. Further, by basing our benchmark in a realistic IoT sensor network scenario, we can combine paradigms such as low-latency event processing, machine learning inference, and offline data analytics, and analyze the performance impact of their interplay in a fog data processing system

Skills and Knowledge for Data-Intensive Environmental Research.

The scale and magnitude of complex and pressing environmental issues lend urgency to the need for integrative and reproducible analysis and synthesis, facilitated by data-intensive research approaches. However, the recent pace of technological change has been such that appropriate skills to accomplish data-intensive research are lacking among environmental scientists, who more than ever need greater access to training and mentorship in computational skills. Here, we provide a roadmap for raising data competencies of current and next-generation environmental researchers by describing the concepts and skills needed for effectively engaging with the heterogeneous, distributed, and rapidly growing volumes of available data. We articulate five key skills: (1) data management and processing, (2) analysis, (3) software skills for science, (4) visualization, and (5) communication methods for collaboration and dissemination. We provide an overview of the current suite of training initiatives available to environmental scientists and models for closing the skill-transfer gap