Efficient HTTP based I/O on very large datasets for high performance computing with the libdavix library

Remote data access for data analysis in high performance computing is commonly done with specialized data access protocols and storage systems. These protocols are highly optimized for high throughput on very large datasets, multi-streams, high availability, low latency and efficient parallel I/O. The purpose of this paper is to describe how we have adapted a generic protocol, the Hyper Text Transport Protocol (HTTP) to make it a competitive alternative for high performance I/O and data analysis applications in a global computing grid: the Worldwide LHC Computing Grid. In this work, we first analyze the design differences between the HTTP protocol and the most common high performance I/O protocols, pointing out the main performance weaknesses of HTTP. Then, we describe in detail how we solved these issues. Our solutions have been implemented in a toolkit called davix, available through several recent Linux distributions. Finally, we describe the results of our benchmarks where we compare the performance of davix against a HPC specific protocol for a data analysis use case.Comment: Presented at: Very large Data Bases (VLDB) 2014, Hangzho

Reducing Internet Latency : A Survey of Techniques and their Merit

Bob Briscoe, Anna Brunstrom, Andreas Petlund, David Hayes, David Ros, Ing-Jyh Tsang, Stein Gjessing, Gorry Fairhurst, Carsten Griwodz, Michael WelzlPeer reviewedPreprin

Managing Network Delay for Browser Multiplayer Games

Latency is one of the key performance elements affecting the quality of experience (QoE) in computer games. Latency in the context of games can be defined as the time between the user input and the result on the screen. In order for the QoE to be satisfactory the game needs to be able to react fast enough to player input. In networked multiplayer games, latency is composed of network delay and local delays. Some major sources of network delay are queuing delay and head-of-line (HOL) blocking delay. Network delay in the Internet can be even in the order of seconds. In this thesis we discuss what feasible networking solutions exist for browser multiplayer games. We conduct a literature study to analyze the Differentiated Services architecture, some salient Active Queue Management (AQM) algorithms (RED, PIE, CoDel and FQ-CoDel), the Explicit Congestion Notification (ECN) concept and network protocols for web browser (WebSocket, QUIC and WebRTC). RED, PIE and CoDel as single-queue implementations would be sub-optimal for providing low latency to game traffic. FQ-CoDel is a multi-queue AQM and provides flow separation that is able to prevent queue-building bulk transfers from notably hampering latency-sensitive flows. WebRTC Data-Channel seems promising for games since it can be used for sending arbitrary application data and it can avoid HOL blocking. None of the network protocols, however, provide completely satisfactory support for the transport needs of multiplayer games: WebRTC is not designed for client-server connections, QUIC is not designed for traffic patterns typical for multiplayer games and WebSocket would require parallel connections to mitigate the effects of HOL blocking

A Demonstration Scenario for the NorNet Core Multi-Homed Network Testbed

There is extensive on-going research taking place with the goal of implementing transport layer protocols that are able to utilize multi-homing on machines (connection to multiple ISPs). Multi-homing has potential to improve resilience and increase total goodput compared to using a single connection. This would e.g. benefit the trend of more and more services being moved online and into the cloud. The NorNet Core multi-homed network testbed aims to facilitate this research by offering programmable virtual machines located at multiple different sites across the world, accessible for researchers. The aim of this project is the design and implementation of a demonstration platform for NorNet Core that enables the testbed to be illustratively demonstrated to potential new users. This is done by offering users interactive experiments, and geographical visualization of various communication scenarios between nodes in the NorNet Core network in real time. The demonstration platform puts emphasis on demonstrating the unique aspects of NorNet Core compared to other networking testbeds; namely the multi-homed nature and IPv6 support for all of the nodes. This thesis discusses the technical aspects of designing and implementing the platform, as well as demonstrating the finished result with various demonstration scenarios. The Qt programming framework was chosen as the primary development framework for this application

Transport layer protocols and architectures for satellite networks

Designing efficient transmission mechanisms for advanced satellite networks is a demanding task, requiring the definition and the implementation of protocols and architectures well suited to this challenging environment. In particular, transport protocols performance over satellite networks is impaired by the characteristics of the satellite radio link, specifically by the long propagation delay and the possible presence of segment losses due to physical channel errors. The level of impact on performance depends upon the link design (type of constellation, link margin, coding and modulation) and operational conditions (link obstructions, terminal mobility, weather conditions, etc.). To address these critical aspects a number of possible solutions have been presented in the literature, ranging from limited modifications of standard protocols (e.g. TCP, transmission control protocol) to completely alternative protocol and network architectures. However, despite the great number of different proposals (or perhaps also because of it), the general framework appears quite fragmented and there is a compelling need of an integration of the research competences and efforts. This is actually the intent of the transport protocols research line within the European SatNEx (Satellite Network of Excellence) project. Stemming from the authors' work on this project, this paper aims to provide the reader with an updated overview of all the possible approaches that can be pursued to overcome the limitations of current transport protocols and architectures, when applied to satellite communications. In the paper the possible solutions are classified in the following categories: optimization of TCP interactions with lower layers, TCP enhancements, performance enhancement proxies (PEP) and delay tolerant networks (DTN). Advantages and disadvantages of the different approaches, as well as their interactions, are investigated and discussed, taking into account performance improvement, complexity, and compliance to the standard semantics. From this analysis, it emerges that DTN architectures could integrate some of the most efficient solutions from the other categories, by inserting them in a new rigorous framework. These innovative architectures therefore may represent a promising solution for solving some of the important problems posed at the transport layer by satellite networks, at least in a medium-to-long-term perspective. Copyright (c) 2006 John Wiley & Sons, Ltd