SciKit-GStat 1.0: a SciPy-flavored geostatistical variogram estimation toolbox written in Python

Geostatistical methods are widely used in almost all geoscientific disciplines, i.e., for interpolation, rescaling, data assimilation or modeling. At its core, geostatistics aims to detect, quantify, describe, analyze and model spatial covariance of observations. The variogram, a tool to describe this spatial covariance in a formalized way, is at the heart of every such method. Unfortunately, many applications of geostatistics focus on the interpolation method or the result rather than the quality of the estimated variogram. Not least because estimating a variogram is commonly left as a task for computers, and some software implementations do not even show a variogram to the user. This is a miss, because the quality of the variogram largely determines whether the application of geostatistics makes sense at all. Furthermore, the Python programming language was missing a mature, well-established and tested package for variogram estimation a couple of years ago. Here I present SciKit-GStat, an open-source Python package for variogram estimation that fits well into established frameworks for scientific computing and puts the focus on the variogram before more sophisticated methods are about to be applied. SciKit-GStat is written in a mutable, object-oriented way that mimics the typical geostatistical analysis workflow. Its main strength is the ease of use and interactivity, and it is therefore usable with only a little or even no knowledge of Python. During the last few years, other libraries covering geostatistics for Python developed along with SciKit-GStat. Today, the most important ones can be interfaced by SciKit-GStat. Additionally, established data structures for scientific computing are reused internally, to keep the user from learning complex data models, just for using SciKit-GStat. Common data structures along with powerful interfaces enable the user to use SciKit-GStat along with other packages in established workflows rather than forcing the user to stick to the author\u27s programming paradigms. SciKit-GStat ships with a large number of predefined procedures, algorithms and models, such as variogram estimators, theoretical spatial models or binning algorithms. Common approaches to estimate variograms are covered and can be used out of the box. At the same time, the base class is very flexible and can be adjusted to less common problems, as well. Last but not least, it was made sure that a user is aided in implementing new procedures or even extending the core functionality as much as possible, to extend SciKit-GStat to uncovered use cases. With broad documentation, a user guide, tutorials and good unit-test coverage, SciKit-GStat enables the user to focus on variogram estimation rather than implementation details

An Ensemble Approach to Space-Time Interpolation

There has been much excitement and activity in recent years related to the relatively sudden availability of earth-related data and the computational capabilities to visualize and analyze these data. Despite the increased ability to collect and store large volumes of data, few individual data sets exist that provide both the requisite spatial and temporal observational frequency for many urban and/or regional-scale applications. The motivating view of this paper, however, is that the relative temporal richness of one data set can be leveraged with the relative spatial richness of another to fill in the gaps. We also note that any single interpolation technique has advantages and disadvantages. Particularly when focusing on the spatial or on the temporal dimension, this means that different techniques are more appropriate than others for specific types of data. We therefore propose a space- time interpolation approach whereby two interpolation methods – one for the temporal and one for the spatial dimension – are used in tandem in order to maximize the quality of the result. We call our ensemble approach the Space-Time Interpolation Environment (STIE). The primary steps within this environment include a spatial interpolator, a time-step processor, and a calibration step that enforces phenomenon-related behavioral constraints. The specific interpolation techniques used within the STIE can be chosen on the basis of suitability for the data and application at hand. In the current paper, we describe STIE conceptually including the structure of the data inputs and output, details of the primary steps (the STIE processors), and the mechanism for coordinating the data and the 1 processors. We then describe a case study focusing on urban land cover in Phoenix Arizona. Our empirical results show that STIE was effective as a space-time interpolator for urban land cover with an accuracy of 85.2% and furthermore that it was more effective than a single technique.

An Ensemble Approach to Space-Time Interpolation

There has been much excitement and activity in recent years related to the relatively sudden availability of earth-related data and the computational capabilities to visualize and analyze these data. Despite the increased ability to collect and store large volumes of data, few individual data sets exist that provide both the requisite spatial and temporal observational frequency for many urban and/or regional-scale applications. The motivating view of this paper, however, is that the relative temporal richness of one data set can be leveraged with the relative spatial richness of another to fill in the gaps. We also note that any single interpolation technique has advantages and disadvantages. Particularly when focusing on the spatial or on the temporal dimension, this means that different techniques are more appropriate than others for specific types of data. We therefore propose a space- time interpolation approach whereby two interpolation methods – one for the temporal and one for the spatial dimension – are used in tandem in order to maximize the quality of the result. We call our ensemble approach the Space-Time Interpolation Environment (STIE). The primary steps within this environment include a spatial interpolator, a time-step processor, and a calibration step that enforces phenomenon-related behavioral constraints. The specific interpolation techniques used within the STIE can be chosen on the basis of suitability for the data and application at hand. In the current paper, we describe STIE conceptually including the structure of the data inputs and output, details of the primary steps (the STIE processors), and the mechanism for coordinating the data and the processors. We then describe a case study focusing on urban land cover in Phoenix, Arizona. Our empirical results show that STIE was effective as a space-time interpolator for urban land cover with an accuracy of 85.2% and furthermore that it was more effective than a single technique.

Using choice experiments to explore the spatial distribution of willingness to pay for rural landscape improvements

We report findings from a choice experiment survey designed to estimate the economic benefits from policy measures which improve the rural landscape under an agri-environment scheme in the Republic of Ireland. Using a panel mixed logit specification to account for unobserved taste heterogeneity we derive individual- specific willingness to pay estimates for each respondent in the sample. We subsequently investigate the existence of spatial dependence of these estimates. Results suggest the existence of positive spatial autocorrelation for all rural landscape attributes. As a means of benefit transfer, kriging methods are employed to interpolate willingness to pay estimates across the whole of the Republic of Ireland. The kriged WTP surfaces confirm the existence of spatial dependence and illustrate the implied spatial variation and regional disparities in WTP for all the rural landscape improvements

Methods in machine learning for probabilistic modelling of environment, with applications in meteorology and geology

Earth scientists increasingly deal with ‘big data’. Where once we may have struggled to obtain a handful of relevant measurements, we now often have data being collected from multiple sources, on the ground, in the air, and from space. These observations are accumulating at a rate that far outpaces our ability to make sense of them using traditional methods with limited scalability (e.g., mental modelling, or trial-and-error improvement of process based models). The revolution in machine learning offers a new paradigm for modelling the environment: rather than focusing on tweaking every aspect of models developed from the top down based largely on prior knowledge, we now have the capability to instead set up more abstract machine learning systems that can ‘do the tweaking for us’ in order to learn models from the bottom up that can be considered optimal in terms of how well they agree with our (rapidly increasing number of) observations of reality, while still being guided by our prior beliefs. In this thesis, with the help of spatial, temporal, and spatio-temporal examples in meteorology and geology, I present methods for probabilistic modelling of environmental variables using machine learning, and explore the considerations involved in developing and adopting these technologies, as well as the potential benefits they stand to bring, which include improved knowledge-acquisition and decision-making. In each application, the common theme is that we would like to learn predictive distributions for the variables of interest that are well-calibrated and as sharp as possible (i.e., to provide answers that are as precise as possible while remaining honest about their uncertainty). Achieving this requires the adoption of statistical approaches, but the volume and complexity of data available mean that scalability is an important factor — we can only realise the value of available data if it can be successfully incorporated into our models.Engineering and Physical Sciences Research Council (EPSRC

Des observations à la construction d'un modèle de simulation de la dynamique urbaine : une approche inductive

International audienceThrough this paper, we chose to present a modelling concept the aim of which is the simulation of the intra-urban location dynamics. This concept includes some characteristics which relate it to the synergetic models as well as to the micro-simulation ones. In the first part of this paper, we describe and examine the modelling concept to focus in a second part on a more detailed presentation of one aspect of the model concerning the measure of the attractiveness of the neighbourhoods of a town.Dans cet article, nous avons choisi de présenter un concept de modélisation dont le but est la simulation de la dynamique des localisations d'activités en milieu intra-urbain. De par ses caractéristiques, ce concept se rattache tant aux modèles synergétiques qu'à ceux de micro-simulation. Dans la première partie de cet article, nous décrivons et commentons le concept de modélisation proposé pour, dans une seconde partie, s'attacher plus particulièrement à la présentation détaillée d'un aspect du modèle, à savoir la mesure de l'attractivité des quartiers d'une ville

3D and 4D Simulations for Landscape Reconstruction and Damage Scenarios. GIS Pilot Applications

The project 3D and 4D Simulations for Landscape Reconstruction and Damage Scenarios: GIS Pilot Applications has been devised with the intention to deal with the demand for research, innovation and applicative methodology on the part of the international programme, requiring concrete results to increase the capacity to know, anticipate and respond to a natural disaster. This project therefore sets out to develop an experimental methodology, a wide geodatabase, a connected performant GIS platform and multifunctional scenarios able to profitably relate the added values deriving from different geotechnologies, aimed at a series of crucial steps regarding landscape reconstruction, event simulation, damage evaluation, emergency management, multi-temporal analysis. The Vesuvius area has been chosen for the pilot application owing to such an impressive number of people and buildings subject to volcanic risk that one could speak in terms of a possible national disaster. The steps of the project move around the following core elements: creation of models that reproduce the territorial and anthropic structure of the past periods, and reconstruction of the urbanized area, with temporal distinctions; three-dimensional representation of the Vesuvius area in terms of infrastructuralresidential aspects; GIS simulation of the expected event; first examination of the healthcareepidemiological consequences; educational proposals. This paper represents a proactive contribution which describes the aims of the project, the steps which constitute a set of specific procedures for the methodology which we are experimenting, and some thoughts regarding the geodatabase useful to “package” illustrative elaborations. Since the involvement of the population and adequate hazard preparedness are very important aspects, some educational and communicational considerations are presented in connection with the use of geotechnologies to promote the knowledge of risk

Practicable methodologies for delivering comprehensive spatial soils information

This thesis is concerned with practicable methodologies for delivering comprehensive spatial soil information to end-users. There is a need for relevant spatial soil information to complement objective decision-making for addressing current problems associated with soil degradation; for modelling, monitoring and measurement of particular soil services; and for the general management of soil resources. These are real-world situations, which operate at spatial scales ranging from field to global scales. As such, comprehensive spatial soil information is tailored to meet the spatial scale specifications of the end user, and is of a nature that fully characterises the whole-soil profile with associated prediction uncertainties, and where possible, both the predictions and uncertainties have been independently validated. ‘Practicable’ is an idealistic pursuit but nonetheless necessary because of a need to equip land-holders, private-sector and non-governmental stakeholders and, governmental departments including soil mapping agencies with the necessary tools to ensure wide application of the methodologies to match the demand for relevant spatial soil information. Practicable methodologies are general and computationally efficient; can be applied to a wide range of soil attributes; can handle variable qualities of data; and are effective when working with very large datasets. In this thesis, delivering comprehensive spatial soil information relies on coupling legacy soil information (principally site observations made in the field) with Digital Soil Mapping (DSM) which comprises quantitative, state-of-the-art technologies for soil mapping. After the General Introduction, a review of the literature is given in Chapter 1 which describes the research context of the thesis. The review describes soil mapping first from a historical perspective and rudimentary efforts of mapping soils and then tracks the succession of advances that have been made towards the realisation of populated, digital spatial soil information databases where measures of prediction certainties are also expressed. From the findings of the review, in order to deliver comprehensive spatial soil information to end-users, new research was required to investigate: 1) a general method for digital soil mapping the whole-profile (effectively pseudo-3D) distribution of soil properties; 2) a general method for quantifying the total prediction uncertainties of the digital soil maps that describe the whole-profile distribution of soil properties; 3) a method for validating the whole-profile predictions of soil properties and the quantifications of their uncertainties; 4) a systematic framework for scale manipulations or upscaling and downscaling techniques for digital soil mapping as a means of generating soil information products tailored to the needs of soil information users. Chapters 2 to 6 set about investigating how we might go about doing these with a succession of practicable methodologies. Chapter 2 addressed the need for whole-profile mapping of soil property distribution. Equal-area spline depth functions coupled with DSM facilitated continuous mapping the lateral and vertical distribution of soil properties. The spline function is a useful tool for deriving the continuous variation of soil properties from soil profile and core observations and is also suitable to use for a number of different soil properties. Generally, mapping the continuous depth function of soil properties reveals that the accuracy of the models is highest at the soil surface but progressively decreases with increasing soil depth. Chapter 3 complements the investigations made in Chapter 2 where an empirical method of quantifying prediction uncertainties from DSM was devised. This method was applied for quantifying the uncertainties of whole-profile digital soil maps. Prediction uncertainty with the devised empirical method is expressed as a prediction interval of the underlying model errors. The method is practicable in the sense that it accounts for all sources of uncertainty and is computationally efficient. Furthermore the method is amenable in situations where complex spatial soil prediction functions such as regression kriging approaches are used. Proper evaluation of digital soil maps requires testing the predictions and the quantification of the prediction uncertainties. Chapter 4 devised two new criteria in which to properly evaluate digital soil maps when additional soil samples collected by probability sampling are used for validation. The first criterion addresses the accuracy of the predictions in the presence of uncertainties and is the spatial average of the statistical expectation of the Mean Square Error of a simulated random value (MSES). The second criterion addresses the quality of the uncertainties which is estimated as the total proportion of the study area where the (1-α)-prediction interval (PI) covers the true value (APCP). Ideally these criteria will be coupled with conventional measures of map quality so that objective decisions can be made about the reliability and subsequent suitability of a map for a given purpose. It was revealed in Chapter 4, that the quantifications of uncertainty are susceptible to bias as a result of using legacy soil data to construct spatial soil prediction functions. As a consequence, in addition to an increasing uncertainty with soil depth, there is increasing misspecification of the prediction uncertainties. Chapter 2, 3, and 4 thus represent a framework for delivering whole-soil profile predictions of soil properties and their uncertainties, where both have been assessed or validated across mapping domains at a range of spatial scales for addressing field, farm, regional, catchment, national, continental or global soil-related problems. The direction of Chapters 5 and 6 however addresses issues specifically related to tailoring spatial soil information to the scale specifications of the end-user through the use of scale manipulations on existing digital soil maps. What is proposed in Chapter 5 is a scaling framework that takes into account the scaling triplet of digital soil maps—extent, resolution, and support—and recommends pedometric methodologies for scale manipulation based on the scale entities of the source and destination maps. Upscaling and downscaling are descriptors for moving up to coarser or down to finer scales respectively but may be too general for DSM. Subsequently Fine-gridding and coarse-gridding are operations where the grid spacing changes but support remains unchanged. Deconvolution and convolution are operations where the support always changes, which may or may not involve changing the grid spacing. While disseveration and conflation operations occur when the support and grid size are equal and both are then changed equally and simultaneously. There is an increasing richness of data sources describing the physical distribution of the Earth’s resources with improved qualities and resolutions. To take advantage of this, Chapter 6 devises a novel procedure for downscaling, involving disseveration. The method attempts to maintain the mass balance of the fine scaled predictions with the available coarse scaled information, through an iterative algorithm which attempts to reconstruct the variation of a property at a prescribed fine scale through an empirical function using environmental or covariate information. One of the advantages associated with the devised method is that soil property uncertainties at the coarse scale can be incorporated into the downscaling algorithm. Finally Chapter 7 presents a synthesis of the investigations made in Chapters 2 to 6 and summarises the pertinent findings. Directly from the investigations carried out during this project there are opportunities for further work; both in terms of addressing shortcomings that were highlighted but not investigated in the thesis, and more generally for advancing digital soil mapping to an operational status and beyond