4 research outputs found
Accelerating Communications in Federated Applications with Transparent Object Proxies
Advances in networks, accelerators, and cloud services encourage programmers
to reconsider where to compute -- such as when fast networks make it
cost-effective to compute on remote accelerators despite added latency.
Workflow and cloud-hosted serverless computing frameworks can manage multi-step
computations spanning federated collections of cloud, high-performance
computing (HPC), and edge systems, but passing data among computational steps
via cloud storage can incur high costs. Here, we overcome this obstacle with a
new programming paradigm that decouples control flow from data flow by
extending the pass-by-reference model to distributed applications. We describe
ProxyStore, a system that implements this paradigm by providing object proxies
that act as wide-area object references with just-in-time resolution. This
proxy model enables data producers to communicate data unilaterally,
transparently, and efficiently to both local and remote consumers. We
demonstrate the benefits of this model with synthetic benchmarks and real-world
scientific applications, running across various computing platforms