In recent years large amounts of data have been made publicly available: literally thousands of open data sources exist, with genome data, temperature measurements, stock market prices, population and income statistics etc. However, accessing and combining data from different data sources is both non-trivial and very time consuming. These tasks typically take up to 80% of the time of data scientists. Automatic integration and curation of open data can facilitate this process

Cieliebak, Mark

Stadelmann, Thilo

Stockinger, Kurt

ZHAW Zürcher Hochschule für Angewandte Wissenschaften

ERCIM NEWS 100   January 201532Special theme: Scientific Data Sharing and Re-use  Most open data has scientific or govern-mental origins and much of it resides inisolated data stores. In data warehousing,data integration as a discipline providesbest practices concerning data prepara-tion and data management tasks byoffering standardized processes and toolchains. However, with the recent popu-larity of Big Data, an unprecedentednumber of new data sources contribute toan increasingly heterogeneous trove ofdata. Hence, ‘data curation’ – a fullyautomated means of intelligently findingand combining possible data sources inthe unified space of internal and opendata sources – is in high demand [1]. We recently finished a market researchand architectural blueprint, funded bythe Hasler Stiftung, to evaluate require-ments and business cases concerningthe development of such an automaticdata curation system in Switzerland.Market Research The Swiss ICT sector is healthy andinnovative, sustained by strong playersin research (e.g., universities and pri-vately held research institutions) andindustry (e.g., finance and pharma) aswell as a large ecosystem of SMEs andstartups. All surveyed parties amongthis group of stakeholders responded bystating their need for better data cura-tion support of open data that has notbeen previously integrated withininternal data sources. As examples, we identified several usecases that rely on the existence of such aservice: • Economic research would be signifi-cantly improved by using automaticdata curation for unifying tax and rentdata of Swiss municipalities• In a transportation research project,the overall project cost increased by25% because of scattered and hetero-geneous public data. • Scientific digital media archivescould fully index their work, therebycreating new research and applicationpossibilities. For these reasons, national fundingagencies are very keen to support thedevelopment of such a service based ona solid model of business and opera-tions.Such a business model could consist ofoffering automatic data curation as soft-ware-as-a-service for open data. Inorder to comply with open data stan-dards, the access to the data itself has tobe free, while additional services couldbe offered on a freemium basis. To be ofinterest to industrial customers, privateinstallations to curate confidentialinternal data could also be offered.toward Automatic Data Curation for Open Databy Thilo Stadelmann, Mark Cieliebak and Kurt StockingerIn recent years large amounts of data have been made publicly available: literally thousands ofopen data sources exist, with genome data, temperature measurements, stock market prices,population and income statistics etc. However, accessing and combining data from different datasources is both non-trivial and very time consuming. These tasks typically take up to 80% of thetime of data scientists. Automatic integration and curation of open data can facilitate this process.Figure 1: Example ofautomatically integrating fourdifferent data sources for socio-economic research.ERCIM NEWS 100  January 2015 33While academic institutions see theirgeneral mandate to offer such servicesto Swiss researchers, they typically donot target industry use cases. The busi-ness models of current cloud computingproviders do not necessarily coincidewith offering a data curation platform toresearchers, although synergies exist,e.g. through the FIWARE project.Architecture BlueprintWe thus propose SODES – the blueprintfor a Swiss Open Data ExplorationSystem - that enables easy and intuitiveaccess, integration and exploration ofdifferent data sources. Figure 1 showsan example of the effect: Four differentdata sources on Zurich’s Bahnhof-strasse are automatically integratedbased on common ’Time’ and‘Location’ columns despite the differentdata types and granularities therein.SODES is designed as a platform thatoffers content-based search, automaticdata curation and integrated data previewon top of established data technologiessuch as CKAN or Linked Open Data. Asa service on the Internet or within anyorganization, SODES is envisioned toenable data scientists to do  most of theirdata exploration in one place. SODES can be viewed as a wrapperaround existing data portals that focuson collecting resources (see Figure 2):While the user is free to still retrieve thedata directly from the producer, SODESparticularly adds easy-to-use dataupload and data access user interfaces.These are enabled through the semanticcontext comprehension componentSC3, powered by advanced machinelearning. Therefore, SODES mustprocess the data in machine-readableform. For advanced data analytics, usersdownload the combined data to theirtool of choice or access it remotelythrough standard APIs.Future ActivitiesSODES is driven by a consortium ofresearch institutions (Zurich Universityof Applied Sciences, University ofZurich, ETH Zurich), NGOs (Foundationopendata.ch) and industry partners (LiipAG, itopia AG) and targets Swiss data sci-entists in academia and industry. We areopen to additional partners with interestsin operating the service in order to con-tinue development. Links: Swiss Open Data: http://opendata.chFIWARE: http://www.fi-ware.orgReference:[1] M. Stonebraker et al.: “DataCuration at Scale: The Data TamerSystem”, CIDR 2013.Please contact:Mark CieliebakZHAW School of EngineeringTel. +41 58 934 72 39E-mail: ciel@zhaw.chFigure 2: The high-level architecture of SODES in interaction with data providers and data portals.Data analysis is an important topic forseveral domains in computer science,such as data mining, machine learning,data visualization and predictive ana-lytics.  In this context, scientists aim atinventing new techniques and algo-rithms to handle data and identify mean-ingful patterns within datasets.Usually, data analysis techniques areworked out by using benchmarkdatasets: for instance, the UCI MachineLearning Repository contains a lot ofmaterial for different tasks (regressions,prediction, classification, etc.). Thisrepository is widely used; many aca-demic papers in the domain refer to itsdatasets in order to allow meaningfulcomparisons with the state of the art. In practice, preprocessing is often nec-essary to adjust the benchmark data tothe specificity of new algorithms ormethods [1]. More precisely, ‘data pre-processing’ refers to a collection of dif-ferent data transformation techniques:‘cleansing’ (treatment of noise, etc.),‘dimensionality alteration’ (filtering offeatures, etc.) and ‘quantity alteration’(sampling of the data records).Moreover, a preprocessing processcould drastically affect the original data,and the results of a data analysis couldAn Interactive tool for transparent Data Preprocessingby Olivier Parisot and Thomas TamisierWe propose a visual tool to assist data scientists in data preprocessing. The tool interactively shows thetransformation impacts and information loss, while keeping track of the applied preprocessing tasks.

Toward automatic data curation for open data

ZHAW digitalcollection

https://digitalcollection.zhaw.ch/bitstream/11475/5817/1/ERCIM_2015.pdf

Toward automatic data curation for open data

Abstract

Similar works

Full text

Available Versions

ZHAW Zürcher Hochschule für Angewandte Wissenschaften

ZHAW digitalcollection