Towards better utilization of NEXRAD data in hydrology: An overview of hydro-NEXRAD

With a very modest investment in computer hardware and the open-source local data manger (LDM) software from UCAR\u27s Unidata Program Center, an individual researcher can receive a variety of NEXRAD Level III gridded rainfall products, and the unprocessed Level II data in real-time from most NEXRAD radars. Additionally, the National Climatic Data Center has vast archives of these products and Level II data. Still, significant obstacles remain in order to unlock the full potential of the data. One set of obstacles is related to effective management of multi-terabyte data sets: storing, compressing, and backing up. A second set of obstacles, for hydrologists and hydrometeorologists in particular, is that the NEXRAD Level III products are not well suited for application in hydrology. There is a strong need for the generation of high-quality products directly from the Level II data with well-documented steps that include quality control, removal of false echoes, rainfall estimation algorithms with variety of corrections, coordinate conversion and georeferencing, conversion to a convenient data format(s), and integration with GIS. For hydrologists it is imperative that these procedures are basin-centered as opposed to radar-centered. Thirdly, the amount of data present in a multi-year, multi-radar dataset is such that simple cataloging and indexing of the data is not sufficient. Rather, sophisticated metadata extraction and management techniques are required. The authors describe and discuss the Hydro-NEXRAD software system that addresses the above three challenges. With support from the National Science Foundation through its ITR program, the authors are developing a basin-centered framework for addressing all these issues in a comprehensive manner, tailored specifically for use of NEXRAD data in hydrology and hydrometeorology. Through a flexible web interface users can search a large metadata database base, managed by a relational database, for subsets of interest. Well-chosen and documented defaults are provided for the flow from unprocessed NEXRAD data to basin-centered rainfall estimates at a desired space-time resolution. In addition to the web interface, there are web services that provide access to scripts and compiled programs. © 2007 ASCE

Big Data Analytics in Static and Streaming Provenance

Thesis (Ph.D.) - Indiana University, Informatics and Computing,, 2016With recent technological and computational advances, scientists increasingly integrate sensors and model simulations to understand spatial, temporal, social, and ecological relationships at unprecedented scale. Data provenance traces relationships of entities over time, thus providing a unique view on over-time behavior under study. However, provenance can be overwhelming in both volume and complexity; the now forecasting potential of provenance creates additional demands. This dissertation focuses on Big Data analytics of static and streaming provenance. It develops filters and a non-preprocessing slicing technique for in-situ querying of static provenance. It presents a stream processing framework for online processing of provenance data at high receiving rate. While the former is sufficient for answering queries that are given prior to the application start (forward queries), the latter deals with queries whose targets are unknown beforehand (backward queries). Finally, it explores data mining on large collections of provenance and proposes a temporal representation of provenance that can reduce the high dimensionality while effectively supporting mining tasks like clustering, classification and association rules mining; and the temporal representation can be further applied to streaming provenance as well. The proposed techniques are verified through software prototypes applied to Big Data provenance captured from computer network data, weather models, ocean models, remote (satellite) imagery data, and agent-based simulations of agricultural decision making