Developing an open data portal for the ESA climate change initiative

We introduce the rationale for, and architecture of, the European Space Agency Climate Change Initiative (CCI) Open Data Portal (http://cci.esa.int/data/). The Open Data Portal hosts a set of richly diverse datasets – 13 “Essential Climate Variables” – from the CCI programme in a consistent and harmonised form and to provides a single point of access for the (>100 TB) data for broad dissemination to an international user community. These data have been produced by a range of different institutions and vary across both scientific and spatio-temporal characteristics. This heterogeneity of the data together with the range of services to be supported presented significant technical challenges. An iterative development methodology was key to tackling these challenges: the system developed exploits a workflow which takes data that conforms to the CCI data specification, ingests it into a managed archive and uses both manual and automatically generated metadata to support data discovery, browse, and delivery services. It utilises both Earth System Grid Federation (ESGF) data nodes and the Open Geospatial Consortium Catalogue Service for the Web (OGC-CSW) interface, serving data into both the ESGF and the Global Earth Observation System of Systems (GEOSS). A key part of the system is a new vocabulary server, populated with CCI specific terms and relationships which integrates OGC-CSW and ESGF search services together, developed as part of a dialogue between domain scientists and linked data specialists. These services have enabled the development of a unified user interface for graphical search and visualisation – the CCI Open Data Portal Web Presence

NOSQL For Storage and Retrieval of Large LiDAR Data Collections

Developments in LiDAR technology over the past decades have made LiDAR to become a mature and widely accepted source of geospatial information. This in turn has led to an enormous growth in data volume. The central idea for a file-centric storage of LiDAR point clouds is the observation that large collections of LiDAR data are typically delivered as large collections of files, rather than single files of terabyte size. This split of the dataset, commonly referred to as tiling, was usually done to accommodate a specific processing pipeline. It makes therefore sense to preserve this split. A document oriented NoSQL database can easily emulate this data partitioning, by representing each tile (file) in a separate document. The document stores the metadata of the tile. The actual files are stored in a distributed file system emulated by the NoSQL database. We demonstrate the use of MongoDB a highly scalable document oriented NoSQL database for storing large LiDAR files. MongoDB like any NoSQL database allows for queries on the attributes of the document. As a specialty MongoDB also allows spatial queries. Hence we can perform spatial queries on the bounding boxes of the LiDAR tiles. Inserting and retrieving files on a cloud-based database is compared to native file system and cloud storage transfer speed

Obvious: a meta-toolkit to encapsulate information visualization toolkits. One toolkit to bind them all

This article describes “Obvious”: a meta-toolkit that abstracts and encapsulates information visualization toolkits implemented in the Java language. It intends to unify their use and postpone the choice of which concrete toolkit(s) to use later-on in the development of visual analytics applications. We also report on the lessons we have learned when wrapping popular toolkits with Obvious, namely Prefuse, the InfoVis Toolkit, partly Improvise, JUNG and other data management libraries. We show several examples on the uses of Obvious, how the different toolkits can be combined, for instance sharing their data models. We also show how Weka and RapidMiner, two popular machine-learning toolkits, have been wrapped with Obvious and can be used directly with all the other wrapped toolkits. We expect Obvious to start a co-evolution process: Obvious is meant to evolve when more components of Information Visualization systems will become consensual. It is also designed to help information visualization systems adhere to the best practices to provide a higher level of interoperability and leverage the domain of visual analytics

The application of data mining techniques to interrogate Western Australian water catchment data sets

Current environmental challenges such as increasing dry land salinity, waterlogging, eutrophication and high nutrient runoff in south western regions of Western Australia may have both cultural and environmental implications in the near future. Advances in computer science disciplines, more specifically, data mining techniques and geographic information services provide the means to be able to conduct longitudinal climate studies to predict changes in the Water catchment areas of Western Australia. The research proposes to utilise existing spatial data mining techniques in conjunction of modern open-source geospatial tools to interpret trends in Western Australian water catchment land use. This will be achieved through the development of a innovative data mining interrogation tool that measures and validates the effectiveness of data mining methods on a sample water catchment data set from the Peel Harvey region of WA. In doing so, the current and future statistical evaluation on potential dry land salinity trends can be eluded. The interrogation tool will incorporate different modern geospatial data mining techniques to discover meaningful and useful patterns specific to current agricultural problem domain of dry land salinity. Large GIS data sets of the water catchments on Peel-Harvey region have been collected by the state government Shared Land Information Platform in conjunction with the LandGate agency. The proposed tool will provide an interface for data analysis of water catchment data sets by benchmarking measures using the chosen data mining techniques, such as: classical statistical methods, cluster analysis and principal component analysis.The outcome of research will be to establish an innovative data mining instrument tool for interrogating salinity issues in water catchment in Western Australia, which provides a user friendly interface for use by government agencies, such as Department of Agriculture and Food of Western Australia researchers and other agricultural industry stakeholders

Geoinformation, Geotechnology, and Geoplanning in the 1990s

Over the last decade, there have been some significant changes in the geographic information available to support those involved in spatial planning and policy-making in different contexts. Moreover, developments have occurred apace in the technology with which to handle geoinformation. This paper provides an overview of trends during the 1990s in data provision, in the technology required to manipulate and analyse spatial information, and in the domain of planning where applications of computer technology in the processing of geodata are prominent. It draws largely on experience in western Europe, and in the UK and the Netherlands in particular, and suggests that there are a number of pressures for a strengthened role for geotechnology in geoplanning in the years ahead

Research metadata on the web: selected geospatial data and metadata directories

pp. 131-14

Geoscience after IT: Part L. Adjusting the emerging information system to new technology

Coherent development depends on following widely used standards that respect our vast legacy of existing entries in the geoscience record. Middleware ensures that we see a coherent view from our desktops of diverse sources of information. Developments specific to managing the written word, map content, and structured data come together in shared metadata linking topics and information types

Integrating Spatial Data Linkage and Analysis Services in a Geoportal for China Urban Research

Many geoportals are now evolving into online analytical environments, where large amounts of data and various analysis methods are integrated. These spatiotemporal data are often distributed in different databases and exist in heterogeneous forms, even when they refer to the same geospatial entities. Besides, existing open standards lack sufficient expression of the attribute semantics. Client applications or other services thus have to deal with unrelated preprocessing tasks, such as data transformation and attribute annotation, leading to potential inconsistencies. Furthermore, to build informative interfaces that guide users to quickly understand the analysis methods, an analysis service needs to explicitly model the method parameters, which are often interrelated and have rich auxiliary information. This work presents the design of the spatial data linkage and analysis services in a geoportal for China urban research. The spatial data linkage service aggregates multisource heterogeneous data into linked layers with flexible attribute mapping, providing client applications and services with a unified access as if querying a big table. The spatial analysis service incorporates parameter hierarchy and grouping by extending the standard WPS service, and data‐dependent validation in computation components. This platform can help researchers efficiently explore and analyze spatiotemporal data online.Peer Reviewedhttp://deepblue.lib.umich.edu/bitstream/2027.42/110740/1/tgis12084.pd

Improving predictive asthma algorithms with modelled environment data for Scotland: an observational cohort study protocol

Introduction Asthma has a considerable, but potentially, avoidable burden on many populations globally. Scotland has some of the poorest health outcomes from asthma. Although ambient pollution, weather changes and sociodemographic factors have been associated with asthma attacks, it remains unclear whether modelled environment data and geospatial information can improve population-based asthma predictive algorithms. We aim to create the afferent loop of a national learning health system for asthma in Scotland. We will investigate the associations between ambient pollution, meteorological, geospatial and sociodemographic factors and asthma attacks.Methods and Analysis We will develop and implement a secured data governance and linkage framework to incorporate primary care health data, modelled environment data, geospatial population and sociodemographic data. Data from 75 recruited primary care practices (n=500 000 patients) in Scotland will be used. Modelled environment data on key air pollutants at a horizontal resolution of 5 km×5 km at hourly time steps will be generated using the EMEP4UK atmospheric chemistry transport modelling system for the datazones of the primary care practices’ populations. Scottish population census and education databases will be incorporated into the linkage framework for analysis. We will then undertake a longitudinal retrospective observational analysis. Asthma outcomes include asthma hospitalisations and oral steroid prescriptions. Using a nested case–control study design, associations between all covariates will be measured using conditional logistic regression to account for the matched design and to identify suitable predictors and potential candidate algorithms for an asthma learning health system in Scotland.Findings from this study will contribute to the development of predictive algorithms for asthma outcomes and be used to form the basis for our learning health system prototype.Ethics and dissemination The study received National Health Service Research Ethics Committee approval (16/SS/0130) and also obtained permissions via the Public Benefit and Privacy Panel for Health and Social Care in Scotland to access, collate and use the following data sets: population and housing census for Scotland; Scottish education data via the Scottish Exchange of Data and primary care data from general practice Data Custodians. Analytic code will be made available in the open source GitHub website. The results of this study will be published in international peer reviewed journals