Web technologies for environmental big data

Recent evolutions in computing science and web technology provide the environmental community with continuously expanding resources for data collection and analysis that pose unprecedented challenges to the design of analysis methods, workflows, and interaction with data sets. In the light of the recent UK Research Council funded Environmental Virtual Observatory pilot project, this paper gives an overview of currently available implementations related to web-based technologies for processing large and heterogeneous datasets and discuss their relevance within the context of environmental data processing, simulation and prediction. We found that, the processing of the simple datasets used in the pilot proved to be relatively straightforward using a combination of R, RPy2, PyWPS and PostgreSQL. However, the use of NoSQL databases and more versatile frameworks such as OGC standard based implementations may provide a wider and more flexible set of features that particularly facilitate working with larger volumes and more heterogeneous data sources

Hydrological modelling in a "big data" era: a proof of concept of hydrological models as web services

Dealing with the massive increase in global data availability of all sorts is increasingly being known as big data science. Indeed, largely leveraged by the internet, a new resource of data sets emerges that are so large and heterogeneous that they become awkward to work with. New algorithms, methods and models are needed to filter such data to find trends, test hypotheses, make predictions and quantify uncertainties. As a considerable share of the data relate to environmental processes (e.g., satellite images, distributed sensor networks), this evolution provides exciting challenges for environmental sciences, and hydrology in particular. Web-enabled models are a promising approach to process large and distributed data sets, and to provide tailored products for a variety of end-users. It will also allow hydrological models to be used as building blocks in larger earth system simulation systems. However, in order to do so we need to reconsider the ways that hydrological models are built, results are made available, and uncertainties are quantified. We present the results of an experimental proof of concept of a hydrological modelling web-service to process heterogeneous hydrological data sets. The hydrological model itself consists of a set of conceptual model routines implemented with on a common platform. This framework is linked to global and local data sets through web standards provided by the Open Geospatial Consortium, as well as to a web interface that enables an end-user to request stream flow simulations from a self-defined location. In essence, the proof-of-concept can be seen as an implementation of the Models of Everywhere concept introduced by Beven in 2007. Although the setup is operational and effectively simulates stream flow, we identify several bottlenecks for optimal hydrological simulation in a web-context. The major challenges we identify are related to (1) model selection; (2) uncertainty quantification, and (3) user interaction and scenario analysis. Model selection is inherent to hydrological modelling, because of the large spatial and temporal variability of processes, which inhibits the use of one optimal model structure. However, in a web context it becomes paramount that such selection is automatic, yet objective and transparent. Similarly, uncertainty quantification is a mainstream practice in hydrological modelling, but in a web-context uncertainty analysis face unprecedented challenges in terms of tracking uncertainties throughout a possibly geographically distributed workflow, as well as dealing with an extreme heterogeneity of data availability. Lastly, the ability of end-users to interact directly with hydrological models poses specific challenges in terms of mapping user scenarios (e.g., a scenario of land-use change) into the model parameter space for prediction and uncertainty quantification. The setup has been used in several scientific experiments, including the large-scale UK consortium project on an Environmental Virtual Observatory pilot

Towards a grid-enabled simulation framework for nano-CMOS electronics

The electronics design industry is facing major challenges as transistors continue to decrease in size. The next generation of devices will be so small that the position of individual atoms will affect their behaviour. This will cause the transistors on a chip to have highly variable characteristics, which in turn will impact circuit and system design tools. The EPSRC project "Meeting the Design Challenges of Nano-CMOS Electronics" (Nana-CMOS) has been funded to explore this area. In this paper, we describe the distributed data-management and computing framework under development within Nano-CMOS. A key aspect of this framework is the need for robust and reliable security mechanisms that support distributed electronics design groups who wish to collaborate by sharing designs, simulations, workflows, datasets and computation resources. This paper presents the system design, and an early prototype of the project which has been useful in helping us to understand the benefits of such a grid infrastructure. In particular, we also present two typical use cases: user authentication, and execution of large-scale device simulations

e-Social Science and Evidence-Based Policy Assessment : Challenges and Solutions

Peer reviewedPreprin

Panel on future challenges in modeling methodology

This panel paper presents the views of six researchers and practitioners of simulation modeling. Collectively we attempt to address a range of key future challenges to modeling methodology. It is hoped that the views of this paper, and the presentations made by the panelists at the 2004 Winter Simulation Conference will raise awareness and stimulate further discussion on the future of modeling methodology in areas such as modeling problems in business applications, human factors and geographically dispersed networks; rapid model development and maintenance; legacy modeling approaches; markup languages; virtual interactive process design and simulation; standards; and Grid computing

Simulation in manufacturing and business: A review

Copyright @ 2009 Elsevier B.V.This paper reports the results of a review of simulation applications published within peer-reviewed literature between 1997 and 2006 to provide an up-to-date picture of the role of simulation techniques within manufacturing and business. The review is characterised by three factors: wide coverage, broad scope of the simulation techniques, and a focus on real-world applications. A structured methodology was followed to narrow down the search from around 20,000 papers to 281. Results include interesting trends and patterns. For instance, although discrete event simulation is the most popular technique, it has lower stakeholder engagement than other techniques, such as system dynamics or gaming. This is highly correlated with modelling lead time and purpose. Considering application areas, modelling is mostly used in scheduling. Finally, this review shows an increasing interest in hybrid modelling as an approach to cope with complex enterprise-wide systems

Time series data mining for energy prices forecasting: an application to real data

WOS:000406998500064 (Nº de Acesso Web of Science)Recently, at the 119th European Study Group with Industry, the Energy Solutions Operator EDP proposed a challenge concerning electricity prices simulation, not only for risk measures purposes but also for scenario analysis in terms of pricing and strategy. The main purpose was short-term Electricity Price Forecasting (EPF). This analysis is contextualized in the study of time series behavior, in particular multivariate time series, which is considered one of the current challenges in data mining. In this work a short-term EPF analysis making use of vector autoregressive models (VAR) with exogenous variables is proposed. The results show that the multivariate approach using VAR, with the season of the year and the type of day as exogenous variables, yield a model that explains the intra-day and intra-hour dynamics of the hourly prices

From Social Simulation to Integrative System Design

As the recent financial crisis showed, today there is a strong need to gain "ecological perspective" of all relevant interactions in socio-economic-techno-environmental systems. For this, we suggested to set-up a network of Centers for integrative systems design, which shall be able to run all potentially relevant scenarios, identify causality chains, explore feedback and cascading effects for a number of model variants, and determine the reliability of their implications (given the validity of the underlying models). They will be able to detect possible negative side effect of policy decisions, before they occur. The Centers belonging to this network of Integrative Systems Design Centers would be focused on a particular field, but they would be part of an attempt to eventually cover all relevant areas of society and economy and integrate them within a "Living Earth Simulator". The results of all research activities of such Centers would be turned into informative input for political Decision Arenas. For example, Crisis Observatories (for financial instabilities, shortages of resources, environmental change, conflict, spreading of diseases, etc.) would be connected with such Decision Arenas for the purpose of visualization, in order to make complex interdependencies understandable to scientists, decision-makers, and the general public.Comment: 34 pages, Visioneer White Paper, see http://www.visioneer.ethz.c