Optimizing Sample Design for Approximate Query Processing

The rapid increase of data volumes makes sampling a crucial component of modern data management systems. Although there is a large body of work on database sampling, the problem of automatically determine the optimal sample for a given query remained (almost) unaddressed. To tackle this problem the authors propose a sample advisor based on a novel cost model. Primarily designed for advising samples of a few queries specified by an expert, the authors additionally propose two extensions of the sample advisor. The first extension enhances the applicability by utilizing recorded workload information and taking memory bounds into account. The second extension increases the effectiveness by merging samples in case of overlapping pieces of sample advice. For both extensions, the authors present exact and heuristic solutions. Within their evaluation, the authors analyze the properties of the cost model and demonstrate the effectiveness and the efficiency of the heuristic solutions with a variety of experiments

Collaborative Online Learning For Join Processing

Relational binary operators, such as join, are arguably the most costly and frequently used operations in relational data systems. In many join algorithms, the majority of the process time is spent on scanning and attempting to join the parts of the relations that do not satisfy the join condition and do not generate any join results. It causes slow response time, particularly, in interactive and exploratory environment where users would like real-time performance. Current solutions to this problem usually require lengthy and repeating preprocessings, which are costly in general settings and may not be possible to do in some, such as interactive and exploratory workloads. They may support only for a restricted types of joins. In this report, we outline a novel approach for achieving efficient join processing in which a scan operator learns online and during query execution the portions of its underlying relation that may satisfy the join condition and use them for join sooner. We further improve this method by an algorithm in which both scan operators collaboratively learn an efficient join execution strategy. We also show that this approach generalizes traditional and non-learning methods of join. Our empirical studies using standard benchmarks indicate that this approach outperforms similar methods considerably

Sampling Algorithms for Evolving Datasets

Perhaps the most flexible synopsis of a database is a uniform random sample of the data; such samples are widely used to speed up the processing of analytic queries and data-mining tasks, to enhance query optimization, and to facilitate information integration. Most of the existing work on database sampling focuses on how to create or exploit a random sample of a static database, that is, a database that does not change over time. The assumption of a static database, however, severely limits the applicability of these techniques in practice, where data is often not static but continuously evolving. In order to maintain the statistical validity of the sample, any changes to the database have to be appropriately reflected in the sample. In this thesis, we study efficient methods for incrementally maintaining a uniform random sample of the items in a dataset in the presence of an arbitrary sequence of insertions, updates, and deletions. We consider instances of the maintenance problem that arise when sampling from an evolving set, from an evolving multiset, from the distinct items in an evolving multiset, or from a sliding window over a data stream. Our algorithms completely avoid any accesses to the base data and can be several orders of magnitude faster than algorithms that do rely on such expensive accesses. The improved efficiency of our algorithms comes at virtually no cost: the resulting samples are provably uniform and only a small amount of auxiliary information is associated with the sample. We show that the auxiliary information not only facilitates efficient maintenance, but it can also be exploited to derive unbiased, low-variance estimators for counts, sums, averages, and the number of distinct items in the underlying dataset. In addition to sample maintenance, we discuss methods that greatly improve the flexibility of random sampling from a system's point of view. More specifically, we initiate the study of algorithms that resize a random sample upwards or downwards. Our resizing algorithms can be exploited to dynamically control the size of the sample when the dataset grows or shrinks; they facilitate resource management and help to avoid under- or oversized samples. Furthermore, in large-scale databases with data being distributed across several remote locations, it is usually infeasible to reconstruct the entire dataset for the purpose of sampling. To address this problem, we provide efficient algorithms that directly combine the local samples maintained at each location into a sample of the global dataset. We also consider a more general problem, where the global dataset is defined as an arbitrary set or multiset expression involving the local datasets, and provide efficient solutions based on hashing

Efficient Algorithms for Large Scale Network Problems

In recent years, the growing scale of data has renewed our understanding of what is an efficient algorithm and poses many essential challenges for the algorithm designers. This thesis aims to improve our understanding of many algorithmic problems in this context. These include problems in communication complexity, matching theory, and approximate query processing for database systems. We first study the fundamental and well-known question of {SetIntersection} in communication complexity. We give a result that incorporates the error probability as an independent parameter into the classical trade-off between round complexity and communication complexity. We show that any $r$-round protocol that errs with error probability $2^{-E}$ requires $Omega(Ek^{1/r})$ bits of communication. We also give several almost matching upper bounds. In matching theory, we first study several generalizations of the ordinary matching problem, namely the $f$-matching and $f$-edge cover problem. We also consider the problem of computing a minimum weight perfect matching in a metric space with moderate expansion. We give almost linear time approximation algorithms for all these problems. Finally, we study the sample-based join problem in approximate query processing. We present a result that improves our understanding of the effectiveness and limitations in using sampling to approximate join queries and provides a guideline for practitioners in building AQP systems from a theory perspective.PHDComputer Science & EngineeringUniversity of Michigan, Horace H. Rackham School of Graduate Studieshttps://deepblue.lib.umich.edu/bitstream/2027.42/155263/1/hdawei_1.pd

Verbundoperationen über Datenströmen und deren Optimierung unter Verwendung dynamischer Metadaten

Technischer Fortschritt und Vernetzung haben zu einem enormen Zuwachs an kontinuierlich anfallenden Datenströmen geführt, aus denen immer schneller wertvolle Informationen abgeleitet werden sollen. Zur Verarbeitung von kontinuierlichen Anfragen über Datenströme etablieren sich daher Datenstrommanagementsysteme, die hauptsächlich im Hauptspeicher operieren und im Gegensatz zu Datenbanksystemen auf die Verarbeitung kontinuierlicher Anfragen über Datenströmen zugeschnitten sind. Bei der Implementierung dieser neuen Systeme hat sich die Übernahme des Konzeptes der relationalen Operatorgraphen aus der Welt der Datenbanken bewährt, wobei diese allerdings statt bedarfsgesteuert nun kontinuierlich und datengetrieben ausgewertet werden. Durch Zuweisen von Gültigkeitsintervallen zu den Datenstromelementen kann dabei das Problem gelöst werden, mit endlichen Ressourcen potentiell unbegrenzte Datenströme zu verarbeiten. Die Mächtigkeit solcher Systeme hängt wesentlich von der Verfügbarkeit effizienter wohldefinierter Techniken zur Verknüpfung von Informationen aus verschiedenen Datenströmen ab. Ziel dieser Arbeit ist es daher, das für Datenbanken bewährte Konzept des relationalen Verbundes auf die datengetriebene Datenstromverarbeitung mit Gültigkeitsintervallen zu übertragen. Dazu wird die Semantik der Verbundoperation für Datenströme mittels der Schnappschuss-Reduzierung formal auf die des Verbundes in der erweiterten relationalen Algebra zurückgeführt. Es werden mehrere Verbundalgorithmen vorgestellt und gezeigt, dass diese die gewünschte Semantik haben. Durch eine konsequente Parametrisierung der Verfahren bezüglich der Datenstrukturen zur Statusverwaltung werden verschiedenste Typen von Verbundprädikaten effizient unterstützt. Bewährte Techniken der Verbundberechnung mittels verschachtelter Schleifen, Hashing und Indexierung werden dabei für die Datenstromverarbeitung adaptiert. Mit dem temporalen Progressive-Merge-Join wird zudem ein Verfahren vorgestellt, dass den Verbund über Datenströmen mittels einer wertbasierten Sortierung berechnet. Zudem werden für die Verfahren verschiedenste Optimierungen vorgeschlagen, darunter für alle Verfahren die Verallgemeinerung auf mehr als zwei Datenströme. Zur Ermöglichung der automatischen Auswahl und Parametrisierung der Implementierungen anhand ihres prognostizierten Ressourcenverbrauchs werden diese in ein detailliertes Kostenmodell eingebettet. Da bestimmte Metadaten bezüglich der zu verarbeitenden Datenströmen bei der Registrierung kontinuierlicher Anfragen oftmals nicht vorliegen und sich zudem während der langen Laufzeit der Anfragen ändern können, ist es wichtig, jederzeit detaillierte Informationen bezüglich der Datenströme und des Systemverhaltens erheben und gegebenenfalls durch Anpassungen an der Verarbeitungsstrategie darauf reagieren können. Ein wesentlicher Teil der Arbeit ist daher der Fragestellung gewidmet, wie in einem Datenstrommanagementsystem dynamische Metadaten erhoben werden können. Dazu wird ein benutzerfreundliches Rahmenwerk vorgestellt, das es ermöglicht, dynamische Metadaten konsistent und effizient zu erheben. In Experimenten wird die dynamische Metadatenerhebung untersucht und auch für die Evaluation der vorgestellten Verbundoperationen eingesetzt. Zudem wird eine Technik vorgestellt, mit der kontinuierliche Anfragen zur Laufzeit restrukturiert werden können, und deren Anwendbarkeit für die Verbundoptimierung aufgezeigt

A Disk-Based Join with Probabilistic Guarantees

One of the most common operations in analytic query processing is the application of an aggregate function to the result of a relational join. We describe an algorithm for computing the answer to such a query over large, disk-based input tables. The key innovation of our algorithm is that at all times, it provides an online, statistical estimator for the eventual answer to the query, as well as probabilistic confidence bounds. Thus, a user can monitor the progress of the join throughout its execution and stop the join when satisfied with the estimate’s accuracy, or run the algorithm to completion with a total time requirement that is not much longer than other common join algorithms. This contrasts with other online join algorithms, which either do not offer such statistical guarantees or can only offer guarantees so long as the input data can fit into core memory.