Scalable, Data- intensive Network Computation

To enable groups of collaborating researchers at different locations to effectively share large datasets and investigate their spontaneous hypotheses on the fly, we are interested in de- veloping a distributed system that can be easily leveraged by a variety of data intensive applications. The system is composed of (i) a number of best effort logistical depots to en- able large-scale data sharing and in-network data processing, (ii) a set of end-to-end tools to effectively aggregate, manage and schedule a large number of network computations with attendant data movements, and (iii) a Distributed Hash Table (DHT) on top of the generic depot services for scalable data management. The logistical depot is extended by following the end-to-end principles and is modeled with a closed queuing network model. Its performance characteristics are studied by solving the steady state distributions of the model using local balance equations. The modeling results confirm that the wide area network is the performance bottleneck and running concurrent jobs can increase resource utilization and system throughput. As a novel contribution, techniques to effectively support resource demanding data- intensive applications using the ¯ne-grained depot services are developed. These techniques include instruction level scheduling of operations, dynamic co-scheduling of computation and replication, and adaptive workload control. Experiments in volume visualization have proved the effectiveness of these techniques. Due to the unique characteristic of data- intensive applications and our co-scheduling algorithm, a DHT is implemented on top of the basic storage and computation services. It demonstrates the potential of the Logistical Networking infrastructure to serve as a service creation platform

Java message passing interface.

by Wan Lai Man.Thesis (M.Phil.)--Chinese University of Hong Kong, 1998.Includes bibliographical references (leaves 76-80).Abstract also in Chinese.Chapter 1 --- Introduction --- p.1Chapter 1.1 --- Background --- p.1Chapter 1.2 --- Objectives --- p.3Chapter 1.3 --- Contributions --- p.4Chapter 1.4 --- Overview --- p.4Chapter 2 --- Literature Review --- p.6Chapter 2.1 --- Message Passing Interface --- p.6Chapter 2.1.1 --- Point-to-Point Communication --- p.7Chapter 2.1.2 --- Persistent Communication Request --- p.8Chapter 2.1.3 --- Collective Communication --- p.8Chapter 2.1.4 --- Derived Datatype --- p.9Chapter 2.2 --- Communications in Java --- p.10Chapter 2.2.1 --- Object Serialization --- p.10Chapter 2.2.2 --- Remote Method Invocation --- p.11Chapter 2.3 --- Performances Issues in Java --- p.11Chapter 2.3.1 --- Byte-code Interpreter --- p.11Chapter 2.3.2 --- Just-in-time Compiler --- p.12Chapter 2.3.3 --- HotSpot --- p.13Chapter 2.4 --- Parallel Computing in Java --- p.14Chapter 2.4.1 --- JavaMPI --- p.15Chapter 2.4.2 --- Bayanihan --- p.15Chapter 2.4.3 --- JPVM --- p.15Chapter 3 --- Infrastructure --- p.17Chapter 3.1 --- Layered Model --- p.17Chapter 3.2 --- Java Parallel Environment --- p.19Chapter 3.2.1 --- Job Coordinator --- p.20Chapter 3.2.2 --- HostApplet --- p.20Chapter 3.2.3 --- Formation of Java Parallel Environment --- p.21Chapter 3.2.4 --- Spawning Processes --- p.24Chapter 3.2.5 --- Message-passing Mechanism --- p.28Chapter 3.3 --- Application Programming Interface --- p.28Chapter 3.3.1 --- Message Routing --- p.29Chapter 3.3.2 --- Language Binding for MPI in Java --- p.31Chapter 4 --- Programming in JMPI --- p.35Chapter 4.1 --- JMPI Package --- p.35Chapter 4.2 --- Application Startup Procedure --- p.37Chapter 4.2.1 --- MPI --- p.38Chapter 4.2.2 --- JMPI --- p.38Chapter 4.3 --- Example --- p.39Chapter 5 --- Processes Management --- p.42Chapter 5.1 --- Background --- p.42Chapter 5.2 --- Scheduler Model --- p.43Chapter 5.3 --- Load Estimation --- p.45Chapter 5.3.1 --- Cost Ratios --- p.47Chapter 5.4 --- Task Distribution --- p.49Chapter 6 --- Performance Evaluation --- p.51Chapter 6.1 --- Testing Environment --- p.51Chapter 6.2 --- Latency from Java --- p.52Chapter 6.2.1 --- Benchmarking --- p.52Chapter 6.2.2 --- Experimental Results in Computation Costs --- p.52Chapter 6.2.3 --- Experimental Results in Communication Costs --- p.55Chapter 6.3 --- Latency from JMPI --- p.56Chapter 6.3.1 --- Benchmarking --- p.56Chapter 6.3.2 --- Experimental Results --- p.58Chapter 6.4 --- Application Granularity --- p.62Chapter 6.5 --- Scheduling Enable --- p.64Chapter 7 --- Conclusion --- p.66Chapter 7.1 --- Summary of the thesis --- p.66Chapter 7.2 --- Future work --- p.67Chapter A --- Performance Metrics and Benchmark --- p.69Chapter A.1 --- Model and Metrics --- p.69Chapter A.1.1 --- Measurement Model --- p.69Chapter A.1.2 --- Performance Metrics --- p.70Chapter A.1.3 --- Communication Parameters --- p.72Chapter A.2 --- Benchmarking --- p.73Chapter A.2.1 --- Ping --- p.73Chapter A.2.2 --- PingPong --- p.74Chapter A.2.3 --- Collective --- p.74Bibliography --- p.7

Advances in Grid Computing

This book approaches the grid computing with a perspective on the latest achievements in the field, providing an insight into the current research trends and advances, and presenting a large range of innovative research papers. The topics covered in this book include resource and data management, grid architectures and development, and grid-enabled applications. New ideas employing heuristic methods from swarm intelligence or genetic algorithm and quantum encryption are considered in order to explain two main aspects of grid computing: resource management and data management. The book addresses also some aspects of grid computing that regard architecture and development, and includes a diverse range of applications for grid computing, including possible human grid computing system, simulation of the fusion reaction, ubiquitous healthcare service provisioning and complex water systems

Development Journey of QADPZ - A Desktop Grid Computing Platform

In this paper we present QADPZ, an open source system for desktop grid computing, which enables users of a local network or Internet to share resources. QADPZ allows a centralized management and use of the computational resources of idle computers from a network of desktop computers. QADPZ users can submit compute-intensive applications to the system, which are then automatically scheduled for execution. The scheduling is performed according to the hardware and software requirements of the application. Users can later monitor and control the execution of the applications. Each application consists of one or more tasks. Applications can be independent, when the composing tasks do not require any interaction, or parallel, when the tasks communicate with each other during the computation. The paper describes both QADPZ functionality and the process of design and implementation, with focus on requirements, architecture, user interface and security. Some future work ideas are also presented