Shared Memory Computing on Clusters with Symmetric Multiprocessors and System Area Networks

Cashmere is a software distributed shared memory (S-DSM) system designed for clusters of serverclass machines. It is distinguished from most other S-DSM projects by (1) the effective use of fast user-level messaging, as provided by modern system-area networks, and (2) a “two-level ” protocol structure that exploits hardware coherence within multiprocessor nodes. Fast user-level messages change the tradeoffs in coherence protocol design; they allow Cashmere to employ a relatively simple directory-based coherence protocol. Exploiting hardware coherence within SMP nodes improves overall performance when care is taken to avoid interference with inter-node software coherence. We have implemented Cashmere on a Compaq AlphaServer/Memory Channel cluster, an architecture that provides fast user-level messages. Experiments indicate that a one-level version of the Cashmere protocol provides peformance comparable to, or slightly better than, that of lazy release consistency. Comparisons to Compaq’s Shasta protocol also suggest that while fast user-level messages make finer-grain software DSMs competitive, VM-based systems continue to outperform software-based access control for applications without extensive fine grain sharing. Within the family of Cashmere protocols we find that leveraging intra-node hardware coherenc