A Taxonomy of Data Grids for Distributed Data Sharing, Management and Processing

Data Grids have been adopted as the platform for scientific communities that need to share, access, transport, process and manage large data collections distributed worldwide. They combine high-end computing technologies with high-performance networking and wide-area storage management techniques. In this paper, we discuss the key concepts behind Data Grids and compare them with other data sharing and distribution paradigms such as content delivery networks, peer-to-peer networks and distributed databases. We then provide comprehensive taxonomies that cover various aspects of architecture, data transportation, data replication and resource allocation and scheduling. Finally, we map the proposed taxonomy to various Data Grid systems not only to validate the taxonomy but also to identify areas for future exploration. Through this taxonomy, we aim to categorise existing systems to better understand their goals and their methodology. This would help evaluate their applicability for solving similar problems. This taxonomy also provides a "gap analysis" of this area through which researchers can potentially identify new issues for investigation. Finally, we hope that the proposed taxonomy and mapping also helps to provide an easy way for new practitioners to understand this complex area of research.Comment: 46 pages, 16 figures, Technical Repor

Methods and design issues for next generation network-aware applications

Networks are becoming an essential component of modern cyberinfrastructure and this work describes methods of designing distributed applications for high-speed networks to improve application scalability, performance and capabilities. As the amount of data generated by scientific applications continues to grow, to be able to handle and process it, applications should be designed to use parallel, distributed resources and high-speed networks. For scalable application design developers should move away from the current component-based approach and implement instead an integrated, non-layered architecture where applications can use specialized low-level interfaces. The main focus of this research is on interactive, collaborative visualization of large datasets. This work describes how a visualization application can be improved through using distributed resources and high-speed network links to interactively visualize tens of gigabytes of data and handle terabyte datasets while maintaining high quality. The application supports interactive frame rates, high resolution, collaborative visualization and sustains remote I/O bandwidths of several Gbps (up to 30 times faster than local I/O). Motivated by the distributed visualization application, this work also researches remote data access systems. Because wide-area networks may have a high latency, the remote I/O system uses an architecture that effectively hides latency. Five remote data access architectures are analyzed and the results show that an architecture that combines bulk and pipeline processing is the best solution for high-throughput remote data access. The resulting system, also supporting high-speed transport protocols and configurable remote operations, is up to 400 times faster than a comparable existing remote data access system. Transport protocols are compared to understand which protocol can best utilize high-speed network connections, concluding that a rate-based protocol is the best solution, being 8 times faster than standard TCP. An HD-based remote teaching application experiment is conducted, illustrating the potential of network-aware applications in a production environment. Future research areas are presented, with emphasis on network-aware optimization, execution and deployment scenarios

Transform-domain analysis of packet delay in network nodes with QoS-aware scheduling

In order to differentiate the perceived QoS between traffic classes in heterogeneous packet networks, equipment discriminates incoming packets based on their class, particularly in the way queued packets are scheduled for further transmission. We review a common stochastic modelling framework in which scheduling mechanisms can be evaluated, especially with regard to the resulting per-class delay distribution. For this, a discrete-time single-server queue is considered with two classes of packet arrivals, either delay-sensitive (1) or delay-tolerant (2). The steady-state analysis relies on the use of well-chosen supplementary variables and is mainly done in the transform domain. Secondly, we propose and analyse a new type of scheduling mechanism that allows precise control over the amount of delay differentiation between the classes. The idea is to introduce N reserved places in the queue, intended for future arrivals of class 1

Branch Prediction For Network Processors

Originally designed to favour flexibility over packet processing performance, the future of the programmable network processor is challenged by the need to meet both increasing line rate as well as providing additional processing capabilities. To meet these requirements, trends within networking research has tended to focus on techniques such as offloading computation intensive tasks to dedicated hardware logic or through increased parallelism. While parallelism retains flexibility, challenges such as load-balancing limit its scope. On the other hand, hardware offloading allows complex algorithms to be implemented at high speed but sacrifice flexibility. To this end, the work in this thesis is focused on a more fundamental aspect of a network processor, the data-plane processing engine. Performing both system modelling and analysis of packet processing functions; the goal of this thesis is to identify and extract salient information regarding the performance of multi-processor workloads. Following on from a traditional software based analysis of programme workloads, we develop a method of modelling and analysing hardware accelerators when applied to network processors. Using this quantitative information, this thesis proposes an architecture which allows deeply pipelined micro-architectures to be implemented on the data-plane while reducing the branch penalty associated with these architectures

Parallel and Distributed Computing

The 14 chapters presented in this book cover a wide variety of representative works ranging from hardware design to application development. Particularly, the topics that are addressed are programmable and reconfigurable devices and systems, dependability of GPUs (General Purpose Units), network topologies, cache coherence protocols, resource allocation, scheduling algorithms, peertopeer networks, largescale network simulation, and parallel routines and algorithms. In this way, the articles included in this book constitute an excellent reference for engineers and researchers who have particular interests in each of these topics in parallel and distributed computing

Totally Ordered Broadcast and Multicast Algorithms: A Comprehensive Survey

Total order multicast algorithms constitute an important class of problems in distributed systems, especially in the context of fault-tolerance. In short, the problem of total order multicast consists in sending messages to a set of processes, in such a way that all messages are delivered by all correct destinations in the same order. However, the huge amount of literature on the subject and the plethora of solutions proposed so far make it difficult for practitioners to select a solution adapted to their specific problem. As a result, naive solutions are often used while better solutions are ignored. This paper proposes a classification of total order multicast algorithms based on the ordering mechanism of the algorithms, and describes a set of common characteristics (e.g., assumptions, properties) with which to evaluate them. In this classification, more than fifty total order broadcast and multicast algorithms are surveyed. The presentation includes asynchronous algorithms as well as algorithms based on the more restrictive synchronous model. Fault-tolerance issues are also considered as the paper studies the properties and behavior of the different algorithms with respect to failures

Online learning on the programmable dataplane

This thesis makes the case for managing computer networks with datadriven methods automated statistical inference and control based on measurement data and runtime observations—and argues for their tight integration with programmable dataplane hardware to make management decisions faster and from more precise data. Optimisation, defence, and measurement of networked infrastructure are each challenging tasks in their own right, which are currently dominated by the use of hand-crafted heuristic methods. These become harder to reason about and deploy as networks scale in rates and number of forwarding elements, but their design requires expert knowledge and care around unexpected protocol interactions. This makes tailored, per-deployment or -workload solutions infeasible to develop. Recent advances in machine learning offer capable function approximation and closed-loop control which suit many of these tasks. New, programmable dataplane hardware enables more agility in the network— runtime reprogrammability, precise traffic measurement, and low latency on-path processing. The synthesis of these two developments allows complex decisions to be made on previously unusable state, and made quicker by offloading inference to the network. To justify this argument, I advance the state of the art in data-driven defence of networks, novel dataplane-friendly online reinforcement learning algorithms, and in-network data reduction to allow classification of switchscale data. Each requires co-design aware of the network, and of the failure modes of systems and carried traffic. To make online learning possible in the dataplane, I use fixed-point arithmetic and modify classical (non-neural) approaches to take advantage of the SmartNIC compute model and make use of rich device local state. I show that data-driven solutions still require great care to correctly design, but with the right domain expertise they can improve on pathological cases in DDoS defence, such as protecting legitimate UDP traffic. In-network aggregation to histograms is shown to enable accurate classification from fine temporal effects, and allows hosts to scale such classification to far larger flow counts and traffic volume. Moving reinforcement learning to the dataplane is shown to offer substantial benefits to stateaction latency and online learning throughput versus host machines; allowing policies to react faster to fine-grained network events. The dataplane environment is key in making reactive online learning feasible—to port further algorithms and learnt functions, I collate and analyse the strengths of current and future hardware designs, as well as individual algorithms

Novel applications and contexts for the cognitive packet network

Autonomic communication, which is the development of self-configuring, self-adapting, self-optimising and self-healing communication systems, has gained much attention in the network research community. This can be explained by the increasing demand for more sophisticated networking technologies with physical realities that possess computation capabilities and can operate successfully with minimum human intervention. Such systems are driving innovative applications and services that improve the quality of life of citizens both socially and economically. Furthermore, autonomic communication, because of its decentralised approach to communication, is also being explored by the research community as an alternative to centralised control infrastructures for efficient management of large networks. This thesis studies one of the successful contributions in the autonomic communication research, the Cognitive Packet Network (CPN). CPN is a highly scalable adaptive routing protocol that allows for decentralised control in communication. Consequently, CPN has achieved significant successes, and because of the direction of research, we expect it to continue to find relevance. To investigate this hypothesis, we research new applications and contexts for CPN. This thesis first studies Information-Centric Networking (ICN), a future Internet architecture proposal. ICN adopts a data-centric approach such that contents are directly addressable at the network level and in-network caching is easily supported. An optimal caching strategy for an information-centric network is first analysed, and approximate solutions are developed and evaluated. Furthermore, a CPN inspired forwarding strategy for directing requests in such a way that exploits the in-network caching capability of ICN is proposed. The proposed strategy is evaluated via discrete event simulations and shown to be more effective in its search for local cache hits compared to the conventional methods. Finally, CPN is proposed to implement the routing system of an Emergency Cyber-Physical System for guiding evacuees in confined spaces in emergency situations. By exploiting CPN’s QoS capabilities, different paths are assigned to evacuees based on their ongoing health conditions using well-defined path metrics. The proposed system is evaluated via discrete-event simulations and shown to improve survival chances compared to a static system that treats evacuees in the same way.Open Acces

Multicast outing protocols and architectures in mobile ad-hoc wireless networks

The basic philosophy of personal communication services is to provide user-to-user, location independent communication services. The emerging group communication wireless applications, such as multipoint data dissemination and multiparty conferencing tools have made the design and development of efficient multicast techniques in mobile ad-hoc networking environments a necessity and not just a desire. Multicast protocols in mobile adhoc networks have been an area of active research for the past few years. In this dissertation, protocols and architectures for supporting multicast services are proposed, analyzed and evaluated in mobile ad-hoc wireless networks. In the first chapter, the activities and recent advances are summarized in this work-in-progress area by identifying the main issues and challenges that multicast protocols are facing in mobile ad-hoc networking environments and by surveying several existing multicasting protocols. a classification of the current multicast protocols is presented, the functionality of the individual existing protocols is discussed, and a qualitative comparison of their characteristics is provided according to several distinct features and performance parameters. In the second chapter, a novel mobility-based clustering strategy that facilitates the support of multicast routing and mobility management is presented in mobile ad-hoc networks. In the proposed structure, mobile nodes are organized into nonoverlapping clusters which have adaptive variable-sizes according to their respective mobility. The mobility-based clustering (MBC) approach which is proposed uses combination of both physical and logical partitions of the network (i.e. geographic proximity and functional relation between nodes, such as mobility pattern etc.). In the third chapter, an entropy-based modeling framework for supporting and evaluating the stability is proposed in mobile ad-hoc wireless networks. The basic motivations of the proposed modeling approach stem from the commonality observed in the location uncertainty in mobile ad-hoc wireless networks and the concept of entropy. In the fourth chapter, a Mobility-based Hybrid Multicast Routing (MHMR) protocol suitable for mobile ad-hoc networks is proposed. The MHMR uses the MBC algorithm as the underlying structure. The main features that the proposed protocol introduces are the following: a) mobility based clustering and group based hierarchical structure, in order to effectively support the stability and scalability, b) group based (limited) mesh structure and forwarding tree concepts, in order to support the robustness of the mesh topologies which provides limited redundancy and the efficiency of tree forwarding simultaneously, and c) combination of proactive and reactive concepts which provide the low route acquisition delay of proactive techniques and the low overhead of reactive methods. In the fifth chapter, an architecture for supporting geomulticast services with high message delivery accuracy is presented in mobile ad-hoc wireless networks. Geomulticast is a specialized location-dependent multicasting technique, where messages are multicast to some specific user groups within a specific zone. An analytical framework which is used to evaluate the various geomulticast architectures and protocols is also developed and presented. The last chapter concludes the dissertation

Hyperscale Data Processing With Network-Centric Designs

Today’s largest data processing workloads are hosted in cloud data centers. Due to unprecedented data growth and the end of Moore’s Law, these workloads have ballooned to the hyperscale level, encompassing billions to trillions of data items and hundreds to thousands of machines per query. Enabling and expanding with these workloads are highly scalable data center networks that connect up to hundreds of thousands of networked servers. These massive scales fundamentally challenge the designs of both data processing systems and data center networks, and the classic layered designs are no longer sustainable. Rather than optimize these massive layers in silos, we build systems across them with principled network-centric designs. In current networks, we redesign data processing systems with network-awareness to minimize the cost of moving data in the network. In future networks, we propose new interfaces and services that the cloud infrastructure offers to applications and codesign data processing systems to achieve optimal query processing performance. To transform the network to future designs, we facilitate network innovation at scale. This dissertation presents a line of systems work that covers all three directions. It first discusses GraphRex, a network-aware system that combines classic database and systems techniques to push the performance of massive graph queries in current data centers. It then introduces data processing in disaggregated data centers, a promising new cloud proposal. It details TELEPORT, a compute pushdown feature that eliminates data processing performance bottlenecks in disaggregated data centers, and Redy, which provides high-performance caches using remote disaggregated memory. Finally, it presents MimicNet, a fine-grained simulation framework that evaluates network proposals at datacenter scale with machine learning approximation. These systems demonstrate that our ideas in network-centric designs achieve orders of magnitude higher efficiency compared to the state of the art at hyperscale