Beyond a Single Chip: Clusters of Independent Machines
The shared memory multiprocessors discussed earlier in this series share memory directly and communicate through cache coherence hardware. A Cluster takes a different approach entirely: it connects many independent computers, each with its own separate memory, using a network, and coordinates them through explicit Message Passing rather than shared memory access.
Why Message Passing Is Necessary at This Scale
Maintaining hardware cache coherence, discussed earlier in this series, becomes impractical once a system grows to encompass thousands of separate machines. Instead, each machine in a cluster runs its own independent operating system and memory space, and cooperating machines explicitly send and receive messages over the network to share data and coordinate work, trading the programming simplicity of shared memory for the ability to scale to a vastly larger number of independent processing units.
Warehouse-Scale Computers
A Warehouse-Scale Computer (WSC) extends the cluster concept to an entire building filled with tens of thousands of individual machines, functioning collectively as a single enormous computing resource. This is the architectural foundation underlying large cloud computing platforms and major internet services, where a single user request might be handled by coordinating work across many thousands of machines simultaneously.
Why Warehouse-Scale Systems Require Different Design Thinking
At this scale, individual machine failures are not rare exceptions but routine, expected events happening constantly across such a large number of components. Software running at this scale must be designed from the outset to tolerate individual machine failures gracefully, a fundamentally different mindset from designing software for a single reliable machine.
Network Topologies: How Machines Are Connected
The physical and logical arrangement of connections between machines, called Network Topology, significantly affects both performance and cost at large scale.
- A
Bus Topologyconnects all machines to a single shared communication channel, simple but quickly becoming a bottleneck as more machines are added. - A
Ring Topologyconnects each machine to exactly two neighbors, forming a loop, offering better scalability than a bus but requiring messages to potentially pass through several intermediate machines to reach their destination. - A
MeshorTorus Topologyconnects each machine to several nearby neighbors in a grid-like pattern, offering multiple possible paths between any two machines, improving both performance and fault tolerance at the cost of more complex wiring. - A
Fully Connected Topologyconnects every machine directly to every other machine, offering the best possible performance but becoming prohibitively expensive to wire as the number of machines grows.
Why Topology Choice Involves a Genuine Tradeoff
Choosing a network topology is fundamentally about balancing communication performance against physical wiring cost and complexity. Real large-scale systems typically select a topology, such as a mesh or a more sophisticated hierarchical design, that provides good communication performance for the expected workload pattern without the impractical wiring cost of a fully connected network.
Why This Scale of Parallelism Matters
Clusters and warehouse-scale computers represent the far end of the parallelism spectrum introduced throughout this chapter, extending the same fundamental principles of dividing work and coordinating independent processing units, first seen in multicore chips, all the way up to entire buildings of cooperating machines powering the internet services relied upon daily.