Parallel and Distributed Data Management. Introduction

The manipulation and handling of an ever increasing volume of data by current data-intensive applications require novel techniques for e?cient data management. Despite recent advances in every aspect of data management (storage, access, querying, analysis, mining), future applications are expected to scale to even higher degrees, not only in terms of volumes of data handled but also in terms of users and resources, often making use of multiple, pre-existing autonomous, distributed or heterogeneous resources

Topologically Consistent Models for Efficient Big Geo-Spatio-Temporal Data Distribution

Geo-spatio-temporal topology models are likely to become a key concept to check the consistency of 3D (spatial space) and 4D (spatial + temporal space) models for emerging GIS applications such as subsurface reservoir modelling or the simulation of energy and water supply of mega or smart cities. Furthermore, the data management for complex models consisting of big geo-spatial data is a challenge for GIS and geo-database research. General challenges, concepts, and techniques of big geo-spatial data management are presented. In this paper we introduce a sound mathematical approach for a topologically consistent geo-spatio-temporal model based on the concept of the incidence graph. We redesign DB4GeO, our service-based geo-spatio-temporal database architecture, on the way to the parallel management of massive geo-spatial data. Approaches for a new geo-spatio-temporal and object model of DB4GeO meeting the requirements of big geo-spatial data are discussed in detail. Finally, a conclusion and outlook on our future research are given on the way to support the processing of geo-analytics and -simulations in a parallel and distributed system environment

MINIMIZATION OF RESOURCE CONSUMPTION THROUGH WORKLOAD CONSOLIDATION IN LARGE-SCALE DISTRIBUTED DATA PLATFORMS

The rapid increase in the data volumes encountered in many application domains has led to widespread adoption of parallel and distributed data management systems like parallel databases and MapReduce-based frameworks (e.g., Hadoop) in recent years. Use of such parallel and distributed frameworks is expected to accelerate in the coming years, putting further strain on already-scarce resources like compute power, network bandwidth, and energy. To reduce total execution times, there is a trend towards increasing execution parallelism by spreading out data across a large number of machines. However, this often increases the total resource consumption, and especially energy consumption, significantly because of process startup costs and other overheads (e.g., communication overheads). In this dissertation, we develop several data management techniques to minimize resource consumption through workload consolidation. In this dissertation, we introduce a key metric called query span, i.e., number of machines involved in the execution of a query or a job. In order to minimize the per query resource consumption we propose to minimize query span. To that end, we develop several workload-driven data partitioning and replica selection algorithms that attempt to minimize the average query span by exploiting the fact that most distributed environments need to use replication for fault tolerance. Extensive experiments on various datasets show that judicious data placement and replication can dramatically reduce the average query spans resulting in significant reductions in resource consumption. We show our results primarily on two applications, distributed data warehouse system and distributed information retrieval. In the first case, we show that minimizing average query spans can minimize overall resource consumption for a given workload and can also improve the performance of complex analytical queries. In the second case, our approach minimizes the overall search cost as well as effectively trades off search cost with load imbalance. The best case of resource efficiency for any underlying data processing system is achieved when the job or the query can be run efficiently on a single machine (i.e., query span=1). In the final part of dissertation, we discuss an in-memory MapReduce system optimized for performing complex analytics tasks on input data sizes that fit in a single machine's memory. We argue that systems like Hadoop that are designed to operate across a large number of machines are not optimal in performance for small and medium sized complex analytics tasks because of high startup costs, heavy disk activity, and wasteful checkpointing. We have developed a prototype runtime called HONE that is API compatible with standard (distributed) Hadoop. In other words, we can take existing Hadoop code and run it, without modification, on a multi-core shared memory machine. This allows us to take existing Hadoop algorithms and find the most suitable runtime environment for execution on datasets of varying sizes. Overall, in this dissertation, our key contributions in this work include identification of key metric query span and its relationship with overall resource consumption in scale-out architectures. We introduce several workload-aware techniques to optimize this key metric. We go on to demonstrate the effectiveness of query span minimization on different application scenarios. In order to take advantage of scale-up architectures effectively we develop novel in-memory MapReduce system HONE for single machine. Our thorough experiments on real and synthetic datasets demonstrate the efficacy of our proposed approaches

Discovery in Physics

Volume 2 covers knowledge discovery in particle and astroparticle physics. Instruments gather petabytes of data and machine learning is used to process the vast amounts of data and to detect relevant examples efficiently. The physical knowledge is encoded in simulations used to train the machine learning models. The interpretation of the learned models serves to expand the physical knowledge resulting in a cycle of theory enhancement

Optimization-based User Group Management : Discovery, Analysis, Recommendation

User data is becoming increasingly available in multiple domains ranging from phone usage traces to data on the social Web. User data is a special type of data that is described by user demographics (e.g., age, gender, occupation, etc.) and user activities (e.g., rating, voting, watching a movie, etc.) The analysis of user data is appealing to scientists who work on population studies, online marketing, recommendations, and large-scale data analytics. However, analysis tools for user data is still lacking.In this thesis, we believe there exists a unique opportunity to analyze user data in the form of user groups. This is in contrast with individual user analysis and also statistical analysis on the whole population. A group is defined as set of users whose members have either common demographics or common activities. Group-level analysis reduces the amount of sparsity and noise in data and leads to new insights. In this thesis, we propose a user group management framework consisting of following components: user group discovery, analysis and recommendation.The very first step in our framework is group discovery, i.e., given raw user data, obtain user groups by optimizing one or more quality dimensions. The second component (i.e., analysis) is necessary to tackle the problem of information overload: the output of a user group discovery step often contains millions of user groups. It is a tedious task for an analyst to skim over all produced groups. Thus we need analysis tools to provide valuable insights in this huge space of user groups. The final question in the framework is how to use the found groups. In this thesis, we investigate one of these applications, i.e., user group recommendation, by considering affinities between group members.All our contributions of the proposed framework are evaluated using an extensive set of experiments both for quality and performance.Les donn ́ees utilisateurs sont devenue de plus en plus disponibles dans plusieurs do- maines tels que les traces d'usage des smartphones et le Web social. Les donn ́ees util- isateurs, sont un type particulier de donn ́ees qui sont d ́ecrites par des informations socio-d ́emographiques (ex., ˆage, sexe, m ́etier, etc.) et leurs activit ́es (ex., donner un avis sur un restaurant, voter, critiquer un film, etc.). L'analyse des donn ́ees utilisa- teurs int ́eresse beaucoup les scientifiques qui travaillent sur les ́etudes de la population, le marketing en-ligne, les recommandations et l'analyse des donn ́ees `a grande ́echelle. Cependant, les outils d'analyse des donn ́ees utilisateurs sont encore tr`es limit ́es.Dans cette th`ese, nous exploitons cette opportunit ́e et proposons d'analyser les donn ́ees utilisateurs en formant des groupes d'utilisateurs. Cela diff`ere de l'analyse des util- isateurs individuels et aussi des analyses statistiques sur une population enti`ere. Un groupe utilisateur est d ́efini par un ensemble des utilisateurs dont les membres parta- gent des donn ́ees socio-d ́emographiques et ont des activit ́es en commun. L'analyse au niveau d'un groupe a pour objectif de mieux g ́erer les donn ́ees creuses et le bruit dans les donn ́ees. Dans cette th`ese, nous proposons un cadre de gestion de groupes d'utilisateurs qui contient les composantes suivantes: d ́ecouverte de groupes, analyse de groupes, et recommandation aux groupes.La premi`ere composante concerne la d ́ecouverte des groupes d'utilisateurs, c.- `a-d., compte tenu des donn ́ees utilisateurs brutes, obtenir les groupes d'utilisateurs en op- timisantuneouplusieursdimensionsdequalit ́e. Ledeuxi`emecomposant(c.-`a-d., l'analyse) est n ́ecessaire pour aborder le probl`eme de la surcharge de l'information: le r ́esultat d'une ́etape d ́ecouverte des groupes d'utilisateurs peut contenir des millions de groupes. C'est une tache fastidieuse pour un analyste `a ́ecumer tous les groupes trouv ́es. Nous proposons une approche interactive pour faciliter cette analyse. La question finale est comment utiliser les groupes trouv ́es. Dans cette th`ese, nous ́etudions une applica- tion particuli`ere qui est la recommandation aux groupes d'utilisateurs, en consid ́erant les affinit ́es entre les membres du groupe et son ́evolution dans le temps.Toutes nos contributions sont ́evalu ́ees au travers d'un grand nombre d'exp ́erimentations `a la fois pour tester la qualit ́e et la performance (le temps de r ́eponse)

Topic 5: Parallel and distributed data management

The ever-increasing data volumes used to empower contemporary data-intensive applications as well as aggregations of computing systems call for novel approaches and efficient techniques in the management of geographically dispersed data. Despite recent advances, Internet-scale requirements for both applications and underlying systems require effective provisioning, staging,manipulation, continuous maintenance and monitoring of data hosted in multiple, pre-existing autonomous, distributed and often heterogeneous systems. Evidently, the notions of parallelism and concurrent execution at all levels remain key elements in attaining scalability and effective management for nearly-all modern data-intensive applications. Moreover, as underlying computing environments get transformed through the introduction of novel infrastructures, enhanced capacities and extended functionalities, new solutions are sought to cope with these changes. © 2012 Springer-Verlag