Skip to main content
Article thumbnail
Location of Repository

Diskrouter: A flexible infrastructure for high performance large scale data transfers

By George Kola and Miron Livny

Abstract

The steady increase in data sets of scientific applications, the trend towards collaborative research and the emergence of grid computing has created a need to move large quantities of data over wide-area networks. The dynamic nature of network makes it difficult to tune data transfer protocols to use the full bandwidth. Further, data transfers are limited by the bottleneck link and different links become the bottleneck at different times resulting in under-utilization of other network hops. To address these issues, we have designed a flexible infrastructure that uses hierarchical main memory and disk buffering at intermediate points to speed up transfers. The infrastructure supports application-level multicast to reduce network load and enables easy construction of application-level overlay networks to maximize bandwidth. It can perform dynamic protocol tuning, use higher-level knowledge, and is being in real-life to transfer successfully several terabytes of astronomy images and educational research videos. 1

Year: 2003
OAI identifier: oai:CiteSeerX.psu:10.1.1.134.2079
Provided by: CiteSeerX
Download PDF:
Sorry, we are unable to provide the full text but you may find it at the following location(s):
  • http://citeseerx.ist.psu.edu/v... (external link)
  • http://cs.wisc.edu/~kola/paper... (external link)
  • Suggested articles


    To submit an update or takedown request for this paper, please submit an Update/Correction/Removal Request.