Benczúr, András

Grulich, PM

Markl, V

Rabl, T

Sidló, Csaba István

SZTAKI Publication Repository

STREAMLINE -Streamlined Analysis of Data at Rest and Data in MotionPhilipp M. Grulich 1, Tilmann Rabl 1,2, Volker Markl 1,2, Csaba Sidló 3, Andras Benczur 31 German Research Center for Artificial Intelligence (DFKI), 2 TU Berlin, 3 Hungarian Academy ofSciences (MTA SZTAKI)1firstname.lastname@dfki.de, 2firstname.lastname@tu-berlin.de, 3lastname@sztaki.huABSTRACTSTREAMLINE aims for improving the overall workflow ofbig data analytics systems. For this goal, it combines re-search in different areas to reduce the complexity of the workwith data at rest and data in motion in a unified fashion. Asa foundation STREAMLINE offers a uniform programmingmodel on top of Apache Flink, for which it drives innovationsin a wide range of areas, such as interactive data in motionvisualization and advanced window aggregation techniques.1. PROJECT SUMMARYThe STREAMLINE project aims to improve the workflowand usability of current big data analysis systems. Thereforeit provides a uniform system, which is able to handle theanalysis of big data at rest as well as fast data in motion.With this platform, STREAMLINE enables a reduction ofcomplexity, costs, and latency.Traditionally batch- and stream-processing were consid-ered as two very different types of applications, but in thelast years, it has been shown that the most real-world use-cases required systems for both workloads. This forces com-panies to integrate different specialized systems, which leadsnot only to complex system architecturesand introduces main-tenance overhead, it also introduces a high latency to thegeneral data analysis workflow. This is also known as theproblem of system and human latency in big data analysis.Even technologies that are able to combine data in motionand data at rest are currently very complex and difficult todeploy, maintain and use. Beside this many companies havea demand for much more advanced analyses, which are stillhard to implement in current systems.To reduce this complexity STREAMLINE combines re-search and innovations in the areas of distributed systems,data management, and machine learning. Whereby STREAM-LINE’s key goal is to arrive at sustainable innovation bytechnology transfer to an established and growing open sourceproject. STREAMLINE focuses on innovations in the areaof the following four reactive and proactive applications:c©2017, Copyright is with the authors. Published in Proc. 20th Inter-national Conference on Extending Database Technology (EDBT), March21-24, 2017 - Venice, Italy: ISBN 978-3-89318-073-8, on OpenProceed-ings.org. Distribution of this paper is permitted under the terms of the Cre-ative Commons license CC-by-nc-nd 4.0customer retention, personalized recommendations, targetadvertisement and multilingual Web processing. To inte-grate the innovations into the industry, STREAMLINE part-ners with multiple companies.As its system foundation STREAMLINE relies on theopen source data processing system Apache Flink, whichis able to handle batch and stream processing on a singlepipelined execution engine [1]. On top of this STREAM-LINE offers a single uniform programming model that canautomatically be optimized, parallelized, and adopted to thesystem load, data distribution, and architecture.Research Highlights: Cutty [2] introduces a general ag-gregation sharing framework for streaming windows, whichoutperforms previous solutions in order of magnitudes. Thistechnique utilizes the fact that window aggregations areone of the most redundancy-prone operations in currentstream processing. Cutty is also suitable for multi queryaggregation sharing and non-periodic windows, such as ses-sion window, which can be used for more complex busi-ness logic. Based on this technique STREAMLINE enableshigher throughput and improves the efficiency of its dataprocessing platform.I2 [3] in contrast, focuses on the visualization and inter-active aggregation of data in motion, which is a key enablerfor fast and efficient real-time data analysis. It contributesan interactive development environment, which coordinatesthe cluster application and includes interactive stream visu-alization techniques. With this I2 is able to handle advancedand adaptive aggregations directly on the cluster. As one ex-ample we provide an aggregation algorithm for timer-seriesdata, which reduces the amount of data in a data-rate in-dependent manner and is proven to be correct and minimalin terms of transferred data. Therefore I2 is an importantpart of STREAMLINE, because it enhances the usabilityand accessibility of its platform.2. ACKNOWLEDGEMENTSThis work was supported by the EU Horizon 2020 projectStreamline (688191).3. REFERENCES[1] Carbone et al. Apache flink: Stream and batchprocessing in a single engine. IEEE Data Eng. Bull.,38(4), 2015.[2] Carbone et al. Cutty: Aggregate sharing foruser-defined windows. In CIKM, pages 1201–1210, 2016.[3] Traub et al. I2: Interactive real-time visualization forstreaming data. In EDBT, EDBT, 2017.

STREAMLINE - Streamlined analysis of data at rest and data in motion

Abstract

Similar works

Full text

Available Versions

SZTAKI Publication Repository