Methods and design issues for next generation network-aware applications

Networks are becoming an essential component of modern cyberinfrastructure and this work describes methods of designing distributed applications for high-speed networks to improve application scalability, performance and capabilities. As the amount of data generated by scientific applications continues to grow, to be able to handle and process it, applications should be designed to use parallel, distributed resources and high-speed networks. For scalable application design developers should move away from the current component-based approach and implement instead an integrated, non-layered architecture where applications can use specialized low-level interfaces. The main focus of this research is on interactive, collaborative visualization of large datasets. This work describes how a visualization application can be improved through using distributed resources and high-speed network links to interactively visualize tens of gigabytes of data and handle terabyte datasets while maintaining high quality. The application supports interactive frame rates, high resolution, collaborative visualization and sustains remote I/O bandwidths of several Gbps (up to 30 times faster than local I/O). Motivated by the distributed visualization application, this work also researches remote data access systems. Because wide-area networks may have a high latency, the remote I/O system uses an architecture that effectively hides latency. Five remote data access architectures are analyzed and the results show that an architecture that combines bulk and pipeline processing is the best solution for high-throughput remote data access. The resulting system, also supporting high-speed transport protocols and configurable remote operations, is up to 400 times faster than a comparable existing remote data access system. Transport protocols are compared to understand which protocol can best utilize high-speed network connections, concluding that a rate-based protocol is the best solution, being 8 times faster than standard TCP. An HD-based remote teaching application experiment is conducted, illustrating the potential of network-aware applications in a production environment. Future research areas are presented, with emphasis on network-aware optimization, execution and deployment scenarios

Control plane routing in photonic networks

The work described in the thesis investigates the features of control plane functionality for routing wavelength paths to serve a set of sub-wavelength demands. The work takes account of routing problems only found in physical network layers, notably analogue transmission impairments. Much work exists on routing connections for dynamic Wavelength-Routed Optical Networks (WRON) and to demonstrate their advantages over static photonic networks. However, the question of how agile the WRON should be has not been addressed quantitatively. A categorization of switching speeds is extended, and compared with the reasons for requiring network agility. The increase of effective network capacity achieved with increased agility is quantified through new simulations. It is demonstrated that this benefit only occurs within a certain window of network fill; achievement of significant gain from a more-agile network may be prevented by the operator’s chosen tolerable blocking probability. The Wavelength Path Sharing (WPS) scheme uses semi-static wavelengths to form unidirectional photonic shared buses, reducing the need for photonic agility. Making WPS more practical, novel improved routing algorithms are proposed and evaluated for both execution time and performance, offering significant benefit in speed at modest cost in efficiency. Photonic viability is the question of whether a path that the control plane can configure will work with an acceptable bit error rate (BER) despite the physical transmission impairments encountered. It is shown that, although there is no single approach that is simple, quick to execute and generally applicable at this time, under stated conditions approximations may be made to achieve a general solution that will be fast enough to enable some applications of agility. The presented algorithms, analysis of optimal network agility and viability assessment approaches can be applied in the analysis and design of future photonic control planes and network architectures

Dynamic photonic lightpaths in the StarPlane network

The StarPlane project enables users to dynamically control network photonic paths. Applications running on the Distributed ASCI Supercomputer (DAS-3) can manipulate wavelengths in the Dutch research and education network SURFnet6. The goal is to achieve fast switching times so that when the computational pattern in the computing clusters changes, the underlying network topology adapts rapidly to the new traffic flow. StarPlane develops: the software to perform optimal traffic engineering; the management plane to map the users’ request into directives for the network control plane, as well as the integration to the DAS-3 Grid middleware. We present here some preliminary results obtained with few selected Grid applications that make use of StarPlane