Efficient Employment of Large Format Sensor Data Transfer Architectures

Due to the increasing quantity of data collected by Air Force intelligence, surveillance and reconnaissance (ISR) assets and the focus on timely access to the data collected by these systems, operational data transfer network architectures have become a critical component of their employment in the intelligence production process. Efficient utilization of the provided long-haul communications component of the ISR system improves the value of the single asset to the warfighter and enables connectivity of additional assets via the data transfer network architecture. This research effort focused on the creation and implementation of a structured test design methodology based on the principles of Design of Experiments to propose recommendations for optimization of one such operational architecture while avoiding the common pitfalls of inadequate and inefficient test design and implementation. Factors that could influence the performance of the data transfer network architecture were researched and evaluated to recommend the factors of interest that most greatly affect the efficiency of the operational architecture. To support this evaluation, an emulated network testbed was utilized to develop a representative model of system efficiency. The results of this model indicate that increased aggressiveness for data transfer leads to decreased efficiency in the attempt to utilize available network resources, especially in realm of operations under study that represent non-traditional bandwidth delay product (BDP) networks where network delay is the dominating factor in the determination of BDP. The analysis documented a baseline model of system performance that will be used to guide ongoing maintenance, sustainment and enhancement efforts for the current data transfer capability and provides insight into the recommended test design process for use in development and deployment of future capabilities. The ability to model system performance through the use of a structured and straight-forward process allows for the inclusion of the test design and analysis process in software design and development, as well as, system deployment and operations improvements

Partial Replica Location And Selection For Spatial Datasets

As the size of scientific datasets continues to grow, we will not be able to store enormous datasets on a single grid node, but must distribute them across many grid nodes. The implementation of partial or incomplete replicas, which represent only a subset of a larger dataset, has been an active topic of research. Partial Spatial Replicas extend this functionality to spatial data, allowing us to distribute a spatial dataset in pieces over several locations. We investigate solutions to the partial spatial replica selection problems. First, we describe and develop two designs for an Spatial Replica Location Service (SRLS), which must return the set of replicas that intersect with a query region. Integrating a relational database, a spatial data structure and grid computing software, we build a scalable solution that works well even for several million replicas. In our SRLS, we have improved performance by designing a R-tree structure in the backend database, and by aggregating several queries into one larger query, which reduces overhead. We also use the Morton Space-filling Curve during R-tree construction, which improves spatial locality. In addition, we describe R-tree Prefetching(RTP), which effectively utilizes the modern multi-processor architecture. Second, we present and implement a fast replica selection algorithm in which a set of partial replicas is chosen from a set of candidates so that retrieval performance is maximized. Using an R-tree based heuristic algorithm, we achieve O(n log n) complexity for this NP-complete problem. We describe a model for disk access performance that takes filesystem prefetching into account and is sufficiently accurate for spatial replica selection. Making a few simplifying assumptions, we present a fast replica selection algorithm for partial spatial replicas. The algorithm uses a greedy approach that attempts to maximize performance by choosing a collection of replica subsets that allow fast data retrieval by a client machine. Experiments show that the performance of the solution found by our algorithm is on average always at least 91% and 93.4% of the performance of the optimal solution in 4-node and 8-node tests respectively

Teraflows over Gigabit WANs with UDT

The TCP transport protocol is currently ine#cient for high speed data transfers over long distance networks with high bandwidth delay products. The challenge is to develop a protocol which is fast over networks with high bandwidth delay products, fair to other high volume data streams, and friendly to TCP-based flows. We describe here a UDP based application level transport protocol named UDT (UDP based Data Transfer) with these properties and which is designed to support distributed data intensive computing applications. UDT can utilize high bandwidth e#ciently over wide area networks with high bandwidth delay products. Unlike TCP, UDT is fair to flows independently of their round trip times. In addition, UDT is friendly to concurrent TCP flows, which means it can be deployed not only on experimental research networks but also on production networks. To ensure these properties, UDT employs a novel congestion control approach that combines rate based and window based control mechanisms. In this paper, we describe the congestion control algorithms used by UDT and provide some experimental results demonstrating that UDT is fast, fair and friendly