Connecting to the Data-Intensive Future of Scientific Research

In recent years enormous amounts of digital data have become available to scientific researchers. This flood of data is transforming the way scientific research is conducted. Independent researchers are in serious need of tools that will help them managed and preserve the large volumes of data being created in their own labs. Data management will not only help researchers get or keep a handle on their data, it will also help them stay relevant and competitive in increasingly strict funding environments. This paper provides summaries of best practices and case studies of data management that relate to three common data management challenges – multitudinous sensor data, short-term data loss, and digital images. We use a combination of open system solutions such as HydroServer Lite, an open system database for time series data, and proprietary tools such as Adobe Photoshop Lightroom. Each lab may require its own unique suite of tools, but these are becoming numerous and readily available, making it easier to archive and share data with collaborators and to discover and integrate published data sets

A Resource Centric Approach For Advancing Collaboration Through Hydrologic Data And Model Sharing

HydroShare is an online, collaborative system being developed for open sharing of hydrologic data and models. The goal of HydroShare is to enable scientists to easily discover and access hydrologic data and models, retrieve them to their desktop or perform analyses in a distributed computing environment that may include grid, cloud or high performance computing model instances as necessary. Scientists may also publish outcomes (data, results or models) into HydroShare, using the system as a collaboration platform for sharing data, models and analyses. HydroShare is expanding the data sharing capability of the CUAHSI Hydrologic Information System by broadening the classes of data accommodated, creating new capability to share models and model components, and taking advantage of emerging social media functionality to enhance information about and collaboration around hydrologic data and models. One of the fundamental concepts in HydroShare is that of a Resource. All content is represented using a Resource Data Model that separates system and science metadata and has elements common to all resources as well as elements specific to the types of resources HydroShare will support. These will include different data types used in the hydrology community and models and workflows that require metadata on execution functionality. The HydroShare web interface and social media functions are being developed using the Drupal content management system. A geospatial visualization and analysis component enables searching, visualizing, and analyzing geographic datasets. The integrated Rule-Oriented Data System (iRODS) is being used to manage federated data content and perform rule-based background actions on data and model resources, including parsing to generate metadata catalog information and the execution of models and workflows. This presentation will introduce the HydroShare functionality developed to date, describe key elements of the Resource Data Model and outline the roadmap for future development

Web technologies for environmental big data

Recent evolutions in computing science and web technology provide the environmental community with continuously expanding resources for data collection and analysis that pose unprecedented challenges to the design of analysis methods, workflows, and interaction with data sets. In the light of the recent UK Research Council funded Environmental Virtual Observatory pilot project, this paper gives an overview of currently available implementations related to web-based technologies for processing large and heterogeneous datasets and discuss their relevance within the context of environmental data processing, simulation and prediction. We found that, the processing of the simple datasets used in the pilot proved to be relatively straightforward using a combination of R, RPy2, PyWPS and PostgreSQL. However, the use of NoSQL databases and more versatile frameworks such as OGC standard based implementations may provide a wider and more flexible set of features that particularly facilitate working with larger volumes and more heterogeneous data sources

Urban hydroinformatics: past, present and future

This is the author accepted manuscriptHydroinformatics, as an interdisciplinary domain that blurs boundaries between water science, data science and computer science, is constantly evolving and reinventing itself. At the heart of this evolution, lies a continuous process of critical (self) appraisal of the discipline’s past, present and potential for further evolution, that creates a positive feedback loop between legacy, reality and aspirations. The power of this process is attested by the successful story of hydroinformatics thus far, which has arguably been able to mobilize wide ranging research and development and get the water sector more in tune with the digital revolution of the past 30 years. In this context, this paper attempts to trace the evolution of the discipline, from its computational hydraulics origins to its present focus on the complete socio-technical system, by providing at the same time, a functional framework to improve the understanding and highlight the links between different strands of the state-of-art hydroinformatic research and innovation. Building on this state-of-art landscape, the paper then attempts to provide an overview of key developments that are coming up, on the discipline’s horizon, focusing on developments relevant to urban water management, while at the same time, highlighting important legal, ethical and technical challenges that need to be addressed to ensure that the brightest aspects of this potential future are realized. Despite obvious limitations imposed by a single paper’s ability to report on such a diverse and dynamic field, it is hoped that this work contributes to a better understanding of both the current state of hydroinformatics and to a shared vision on the most exciting prospects for the future evolution of the discipline and the water sector it serves

Advancing Cyberinfrastructure for Collaborative Data Sharing and Modeling in Hydrology

Hydrologic research is increasingly data and computationally intensive, and often involves hydrologic model simulation and collaboration among researchers. With the development of cyberinfrastructure, researchers are able to improve the efficiency, impact, and effectiveness of their research by utilizing online data sharing and hydrologic modeling functionality. However, further efforts are still in need to improve the capability of cyberinfrastructure to serve the hydrologic science community. This dissertation first presents the evaluation of a physically based snowmelt model as an alternative to a temperature index model to improve operational water supply forecasts in the Colorado River Basin. Then it presents the design of the functionality to share multidimensional space-time data in the HydroShare hydrologic information system. It then describes a web application developed to facilitate input preparation and model execution of a snowmelt model and the storage of these results in HydroShare. The snowmelt model evaluation provided use cases to evaluate the cyberinfrastructure elements developed. This research explored a new approach to advance operational water supply forecasts and provided potential solutions for the challenges associated with the design and implementation of cyberinfrastructure for hydrologic data sharing and modeling