Passive available bandwidth: Applying self -induced congestion analysis of application-generated traffic

Monitoring end-to-end available bandwidth is critical in helping applications and users efficiently use network resources. Because the performance of distributed systems is intrinsically linked to the performance of the network, applications that have knowledge of the available bandwidth can adapt to changing network conditions and optimize their performance. A well-designed available bandwidth tool should be easily deployable and non-intrusive. While several tools have been created to actively measure the end-to-end available bandwidth of a network path, they require instrumentation at both ends of the path, and the traffic injected by these tools may affect the performance of other applications on the path.;We propose a new passive monitoring system that accurately measures available bandwidth by applying self-induced congestion analysis to traces of application-generated traffic. The Watching Resources from the Edge of the Network (Wren) system transparently provides available bandwidth information to applications without having to modify the applications to make the measurements and with negligible impact on the performance of applications. Wren produces a series of real-time available bandwidth measurements that can be used by applications to adapt their runtime behavior to optimize performance or that can be sent to a central monitoring system for use by other or future applications.;Most active bandwidth tools rely on adjustments to the sending rate of packets to infer the available bandwidth. The major obstacle with using passive kernel-level traces of TCP traffic is that we have no control over the traffic pattern. We demonstrate that there is enough natural variability in the sending rates of TCP traffic that techniques used by active tools can be applied to traces of application-generated traffic to yield accurate available bandwidth measurements.;Wren uses kernel-level instrumentation to collect traces of application traffic and analyzes the traces in the user-level to achieve the necessary accuracy and avoid intrusiveness. We introduce new passive bandwidth algorithms based on the principles of the active tools to measure available bandwidth, investigate the effectiveness of these new algorithms, implement a real-time system capable of efficiently monitoring available bandwidth, and demonstrate that applications can use Wren measurements to adapt their runtime decisions

Appears in Grid 2003 c○2003 IEEE Comparing Passive Network Monitoring of Grid Application Traffic with Active Probes

Distributed applications require timely network measurements so that they can adapt to changing network conditions and make efficient use of grid resources. One of the key issues in obtaining network measurements is the intrusiveness of the measurements themselves—how much network performance is “wasted ” taking the measurements? Our goal is to combine active and passive monitoring techniques to reduce the need for intrusive measurements without sacrificing the accuracy of the measurements. We are developing a bandwidth monitoring tool as part of the Wren network measurement system that will reduce the burden on the network by passively obtaining measurements from existing application traffic whenever possible, instead of actively probing the network. By using passive measurements when an application is running and active measurements when none are running, we can offer accurate, timely available bandwidth measurements while limiting the invasiveness of active probes. We have completed a prototype of the Wren bandwidth monitoring tool and present our preliminary analysis of its performance in this paper. We provide results from passive implementations of several available bandwidth techniques and demonstrate the close quantitative relationship between the results of both active and passive techniques. We have tested our implementation in a cluster, across a campus, and across the Internet using bulk data transfers as well as an adaptive eigenvalue application. Our results with this diverse set of environments and traffic types show promise toward implementing these techniques as measurement services in production environments

Transparent Optimization of Grid Server Selection With Real-Time Passive

Grid services have tremendously simplified the programming challenges in leveraging large-scale distributed computing. At the same time, the increased level of abstraction reduces the opportunities available to the application for optimizing its performance by monitoring the system. In this paper we introduce a monitoring grid services proxy, which transparently monitors network performance and selects between several replica service providers. This approach provides optimized server selection without any modification to or even awareness of the client application or service providers. We describe how we implement the proxy and monitor the available bandwidth to the service providers using the Wren monitoring toolkit. We present analysis indicating that our monitoring has negligible overhead. Finally, we demonstrate the practicality of our approach by optimizing the server selection for INCOGEN’s VIBE, a bioinformatics workflow application that uploads gene sequences for analysis by remote service providers. We would like to thank INCOGEN for allowing us to use their VIBE software toolkit. In particular, w

Challenges of DHT Design for a Public Communications System

Communications systems, encompassing VoIP, IM, and other personal media, present different challenges for P2P environments than other P2P applications. In particular, reliable communication implies that each resource (person) is unique and must be reliably located, without false negatives. The system must function in the presence of NATs, which create non-transitive connectivity, and must be resilient against DoS attacks that attempt to disrupt its routing properties or DoS a particular person. We have designed and implemented a P2P communications system using an extension of the Chord algorithm as a resource location service. In this paper we present the design tradeoffs necessary to meet the requirements of a reliable communications system. In particular, the practical issues of non-transitive routing, NATs used by residential endpoints, and the prevention of DoS attacks are more critical than strict performance metrics in selecting DHT identifier, topology, and routing algorithms. Where a central authority exists, certificates can be stored in the overlay and allow more efficient DHT algorithms to be used, but securing an open network with NATs requires appropriate Node-ID, replica placement, and routing algorithms

Free Network Measurement For Adaptive Virtualized Distributed Computing

An execution environment consisting of virtual machines (VMs) interconnected with a virtual overlay network can use the naturally occurring traffic of an existing, unmodified application running in the VMs to measure the underlying physical network. Based on these characterizations, and characterizations of the application’s own communication topology, the execution environment can optimize the execution of the application using application-independent means such as VM migration and overlay topology changes. In this paper, we demonstrate the feasibility of such free automatic network measurement by fusing the Wren passive monitoring and analysis system with Virtuoso’s virtual networking system. We explain how Wren has been extended to support on-line analysis, and we explain how Virtuoso’s adaptation algorithms have been enhanced to use Wren’s physical network level information to choose VM-to-host mappings, overlay topology, and forwarding rules