Processing data streams with Quality-of-Service (QoS) guarantees is an emerging area in existing streaming applications. Although it is possible to negotiate the result quality and to reserve the required processing resources in advance, it remains a challenge to adapt the DSMS to data stream characteristics which are not known in advance or are difficult to obtain. Within this paper we present the second generation of our QStream DSMS which addresses the above challenge by using a real-time capable operating system environment for resource reservation and by applying an adaptation mechanism if the data stream characteristics change spontaneously

Legler, Thomas

Lehner, Wolfgang

Schmidt, Sven

Schär, Sebastian

English

Technische Universität Dresden: Qucosa

     Dieses Dokument ist eine Zweitveröffentlichung (Postprint) /  This is a self-archiving document (accepted version):         Diese Version ist verfügbar / This version is available on:  https://nbn-resolving.org/urn:nbn:de:bsz:14-qucosa2-867558             Sven Schmidt, Thomas Legler, Sebastian Schär, Wolfgang Lehner  Robust Real-time Query Processing with QStream   Erstveröffentlichung in / First published in: ICMI05: Seventh International Conference on Multimodal Interfaces 2005. Trondheim, 30. August – 2. September 2005, S. 1299–1301. ACM. ISBN 978-1-59593-154-2. Link: https://dl.acm.org/doi/10.5555/1083592.1083756  Robust Real-time Query Processing with QStreamSven Schmidt Thomas Legler Sebastian Schär Wolfgang LehnerDresden University of TechnologyDatabase Technology Groupdbgroup@mail.inf.tu-dresden.deAbstractProcessing data streams with Quality-of-Service (QoS) guarantees is an emerging areain existing streaming applications. Althoughit is possible to negotiate the result qual-ity and to reserve the required processing re-sources in advance, it remains a challenge toadapt the DSMS to data stream characteris-tics which are not known in advance or are dif-ficult to obtain. Within this paper we presentthe second generation of our QStream DSMSwhich addresses the above challenge by usinga real-time capable operating system environ-ment for resource reservation and by applyingan adaptation mechanism if the data streamcharacteristics change spontaneously.1 IntroductionCurrently, many existing Data Stream ManagementSystems (DSMS), such as [2, 3, 5], aim at an efficient,flexible, and sometimes also distributed evaluation ofstanding queries against streaming data. Often, itis very valuable for the user if the DSMS assures acertain Quality-of-Service (QoS) for query processing.For example, such QoS requirements may include up-per bounds for the tuple processing delay or a guar-anteed result data rate. In [5], we presented an ap-proach of running a DSMS on top of a real-time- ca-pable operating system to meet the QoS requirementsmentioned above. Our work was based on the cal-culation and reservation of necessary resources in ad-vance. However, the main assumptions of our real-time DSMS approach were that the data-dependentparameters (e.g. data rate, data distribution) are©2005 Copyright held by the owner/author(s). Publication rights licensed to ACM. This is the author’s version of the work. It is posted here for your personal use. Not for redistribution. The definitive Version of Record was published in Proceedings of the 31st VLDB Conference, Trondheim, Norway, 2005.known a-priori and, moreover, are constant on aver-age. Only a cumulatively limited jitter of processingtime or data amount was allowed and thus, had influ-ence on the resource calculation and reservation pro-cess. It is obvious that these assumptions do not holdfor many application scenarios, and thus, an exten-sion towards a more flexible management regardingthe data-dependent parameters is required. With thissecond QStream prototype, we establish a so-calledadaptation framework on the basis of the real-time-capable and QoS-guaranteeing system core. The ben-efits are twofold: on the one hand, we are able tohandle arbitrary input data streams, whether or notthe stream characteristics are known a-priori. On theother hand, we enable the user to adjust the robust-ness of the DSMS by granting more or less resources.As a result, processing of arbitrary input streams maybe achieved in real-time, but will be interrupted fromtime to time to adapt the system to changes in thestreaming environment.2 Adaptation FrameworkThe general way of resource reservation based on QoSrequirements is illustrated in figure 1. First of all,the QoS requirements are negotiated. Then, if thecalculated resources are available, they are reservedand guaranteed during runtime. With the resourceguarantee, the quality requirements are guaranteed aswell.QoS NegotiationResource AssuranceResource ReservationQoS AssuranceFigure 1: QoS negotiation processFor our work, we make use of a resource calcula-tion model which is extensively described in [1]. Weimplement a standing query as a network of indepen-dently running basic operators. For optimum resourceusage, and in order to enable QoS guarantees, we ad-Final edited form was published in " ICMI05: Seventh International Conference on Multimodal Interfaces 2005", S. 1299–1301, ISBN 978-1-59593-154-2 https://dl.acm.org/doi/10.5555/1083592.10837561 Provided by Sächsische Landesbibliothek, Staats- und Universitätsbibliothek Dresdenjust the ’speed’ of this operator network based on thedata throughput requirements. The data throughputis determined firstly, by the individual period length ofan operator’s run (called frequency) and secondly, bythe amount of data which is processed during one run.For successful operation, a FIFO buffer is establishedbetween each pair of operators.DSMS Steady StateOnce the resource-determining parameters (like inputdata rate, selectivity of filter and join operators, etc.)are known or estimated, we schedule the DSMS op-erators to work in real-time at the presumed networkspeed, thereby achieving the desired data throughputas well as the maximum tuple delay. As this schedul-ing is based on average parameters, these parameters(mainly the data rate, which we will focus on) mayjitter around their average. It is up to the user tolimit the amount of jitter while calculating necessaryresources. Toleration of more jitter leads to higher in-termediate buffer sizes, a longer tuple delay and higherresource requirements. We suppose the DSMS to runin a steady state as long as tolerated jitter is sufficientfor the fluctuating data stream characteristics. Onlywhile being in this steady state, the DSMS runs in real-time and all time-dependent QoS requirements (tupledelay, data throughput) can be met.Unsteady DSMS - IndicationFrom time to time, certain situations require arescheduling of the DSMS operators. Such situationsinclude, for example, changing input data rates as wellas varying filter predicate and join selectivities. Anindicator for a transition into an unsteady state canbe found when the continuous data flow through net-work operators and FIFO buffers cannot be held upany longer due to blocking buffer accesses: operatorsrepeatedly try either to write data into full buffers orto read data from empty buffers. Thereby, not every(small) increase or decrease of the data rate causes abuffer fault: whether our DSMS is able to overcomefluctuations of the input data rate, and thus, stays ina steady state depends mainly on the tolerated jitterincluded in the calculation.Unsteady DSMS - AdaptationIf the DSMS becomes unsteady, the initially negoti-ated QoS requirements cannot be assured any longer.More precisely, this means that once a buffer read faultor a buffer write fault has occurred, the tuple delayguarantees (and thus the data throughput) are consid-ered broken and, hence, we have to adapt the runningDSMS as soon as possible. Thereby, the DSMS’s pro-cessing speed has to be adjusted to the new situation.Our general goal of the adaptation is to bring thesystem back into a steady state. This comprises threesubgoals:• Depending on the direction of the adaptation(faster or slower network), we either have to freesome of the initially occupied resources or, if pos-sible, we have to implement further allocations.• If the new amount of required resources is avail-able, we will be able to reschedule the DSMS towork at optimum speed regarding the data streamparameters.• Based on the new DSMS speed, we are able tonegotiate new QoS requirements with the DSMSuser, and as long as the DSMS stays steady, weare able to give QoS guarantees.Within QStream, we define the robustness of aDSMS by the duration of its steady states. Figure2 illustrates the influence of data stream characteris-tics and generous resource allocation (jitter tolerance)on the robustness.robustnessjitter tolerancestream characteristicsfluctuationFigure 2: Robustness-determining factorsIf the stream characteristics do not change signif-icantly, and if a high amount of jitter was toleratedwhen calculating and reserving the required resources,the system will become very robust to any change ofits real-time behavior, and thus, the time spans of thesteady states are maximal.Generally, it’s up to the users how much resourcesthey are willing to spend for covering jitter withinthe stream characteristics. The more resources theyspend, the more robust the DSMS will run.3 PrototypeOur QStream prototype is implemented on top of theRealtime Application Interface (RTAI, [4]). The over-all architecture is illustrated in figure 3.QueriesDataSensorResultQueryDSMS EngineQoS Requirementsreal−time capable environmentController QoS MonitorAdaptionFigure 3: Architecture of the QStream systemFinal edited form was published in "  ICMI05: Seventh International Conference on Multimodal Interfaces 2005", S. 1299–1301, ISBN 978-1-59593-154-2 https://dl.acm.org/doi/10.5555/1083592.10837562   Provided by Sächsische Landesbibliothek, Staats- und Universitätsbibliothek DresdenThe system’s core for evaluating standing queriesagainst incoming data streams consists of an operatornetwork running stepwise at a fixed speed to achievethe desired data throughput. Within the operator net-work, FIFO buffers are implemented as shared mem-ory areas to hand over the tuples. A monitor com-ponent is a central part of QStream and may be con-nected to each operator or to each point of the dataflow, where it can gather low-level information, suchas operator runtimes and data throughput statistics,as well as high-level information, like filter predicateselectivities or even histograms of the streaming data.Depending on the amount and on the level of detail,this information forms the basis for planning the re-sources of the DSMS. Every time the system is beingtransferred from one steady state into another one, theresources for the future state have to be determinedbased on statistics and current runtime information.The following diagrams show how QStream adaptsto new data characteristics:00.511.522.533.540 200 400 600 800 1000 1200 1400 1600 1800 2000data rate /  kbyte/stime / ssource data rateDSMS data rate, FIFO buffer = 1000 bytes00.511.522.533.540 200 400 600 800 1000 1200 1400 1600 1800 2000data rate /  kbyte/stime / ssource data rateDSMS data rate, FIFO buffer = 3000 bytes00.511.522.533.540 200 400 600 800 1000 1200 1400 1600 1800 2000data rate /  kbyte/stime / ssource data rateDSMS data rate, FIFO buffer = 5000 bytesFigure 4: Data rate adaptation exampleIn figure 4, the adaptation scenarios for three differ-ent values of jitter tolerance (which directly correlateswith the size of the buffer) are illustrated. The largerthe tolerated jitter, the less often the system has toadapt its processing speed, and the longer the steadystates last. The latter comes along with long-lastingand stable QoS guarantees.0.860.880.90.920.940.960.9810 1000 2000 3000 4000 5000 6000 7000robustnesstolerated jitter (FIFO buffer size / bytes)Figure 5: Robustness measureIn figure 5, we define the robustness as the inverseof the number of adaptations per time unit and markit depending on the tolerated jitter. The general ob-servation is that a larger tolerated jitter, and thus ahigher resource consumption, leads to a higher robust-ness of our prototype.With the QStream demonstration, we will show howdata stream processing can be achieved in real-timeeven if the data stream characteristics change overtime. As a quantitative measure, the robustness en-ables us to reason about how steady a DSMS is andhow stable QoS guarantees can be met.References[1] H. Berthold, S. Schmidt, W. Lehner, and C.-J.Hamann. Integrated resource management for datastream systems. In Proc. of the Annual ACM Sym-posium on Applied Computing (SAC’05, March 13-17, Santa Fe (NM), USA), pages 555–562, 2005.[2] Don Carney, Ugur Çetintemel, Alex Rasin, StanZdonik, Mitch Cherniack, and Michael Stone-braker. Operator scheduling in a data stream man-ager. In Proc. of 29th International Conference onVery Large Databases (VLDB’03, September 9-12,Berlin, Germany), pages 838–849, 2003.[3] Charles D. Cranor, Theodore Johnson, OliverSpatscheck, and Vladislav Shkapenyuk. Gigascope:A stream database for network applications. InProc. of the 2003 ACM SIGMOD InternationalConference on Management of Data (SIGMOD2003, June 9-12, San Diego (CA), USA), pages647–651, 2003.[4] Lorenzo Dozio and Paolo Mantegazza. Realtime distributed control systems using rtai. InProc. of the 6th IEEE International Symposium onObject-Oriented Real-Time Distributed Computing(ISORC 2003, May 14-16, Hakodate, Hokkaido,Japan), pages 11–18, 2003.[5] S. Schmidt, H. Berthold, and W. Lehner. Qstream:Deterministic querying of data streams. InProc. of International Conference on Very LargeData Bases (VLDB’04, August 30 - September 3,Toronto, Canada), pages 1365–1368, 2004.Final edited form was published in "  ICMI05: Seventh International Conference on Multimodal Interfaces 2005", S. 1299–1301, ISBN 978-1-59593-154-2 https://dl.acm.org/doi/10.5555/1083592.10837563   Provided by Sächsische Landesbibliothek, Staats- und Universitätsbibliothek Dresden

Robust Real-time Query Processing with QStream

https://tud.qucosa.de/api/qucosa%3A86755/attachment/ATT-0/

Robust Real-time Query Processing with QStream

Abstract

Similar works

Full text

Available Versions

Technische Universität Dresden: Qucosa