Clusters of workstations have long provided a cost-effective, large-scale parallel computing platform. A Software Distributed Shared Memory (SDSM) system simplifies programming on these platforms by presenting the illusion of shared memory. SDSM performance has historically been limited by the high cost of inter-processor communication overhead. Recent hardware trends, such as commodity symmetric multiprocessors (SMPs) and low-latency, remote-memory-access networks, can be used to potentially lower this overhead.

The Cashmere SDSM has been designed for clusters of SMPs connected by a low-latency, remote-memory-access network. The prototype implementation has been built on a cluster of eight AlphaServer SMPs (32 processors) connected by a Compaq Memory Channel remote-memory-write network. In this talk, I will describe Cashmere's novel techniques for leveraging SMP hardware coherence and for coalescing software coherence operations. These techniques are particularly effective, leading to an average improvement of 25% over a Cashmere version that does not leverage the SMP hardware coherence. Also, I will discuss somewhat surprising results concerning the performance impact of the remote-memory-write network.