Fallacy: More Cores Always Means Proportionally More Performance
As established earlier in this series through Amdahl's Law, the portion of a program that cannot be parallelized places a hard ceiling on achievable speedup, regardless of how many cores are added. Doubling the core count does not double real application performance unless the workload's parallelizable fraction is very high and communication overhead remains low.
Fallacy: A GPU Is Always Faster Than a CPU for Parallel Work
As shown earlier in this series through the roofline model comparison, a GPU's advantage depends heavily on a workload's arithmetic intensity and its tolerance for lockstep execution across thread groups. Workloads with significant branching divergence, discussed earlier regarding GPU architecture, or low arithmetic intensity dominated by memory access can perform worse on a GPU than on a well-optimized CPU implementation.
Fallacy: Shared Memory Programming Is Inherently Simpler at Every Scale
While shared memory, discussed earlier in this series, does simplify programming for a modest number of cores, it does not scale indefinitely. Beyond a certain point, the overhead of maintaining cache coherence, discussed earlier in this series, and memory contention make message-passing approaches, used in clusters and warehouse-scale computers discussed later in this series, necessary for continued scalability.
Pitfall: Ignoring Load Imbalance
Even when a workload is theoretically well-suited to parallelization, uneven distribution of work across processors, discussed earlier regarding real-world speedup measurement, means some processors finish early and sit idle while others remain busy, wasting available parallel capacity that careful load balancing could have recovered.
Pitfall: Underestimating Communication Overhead
As a system scales from a single chip to a cluster to a warehouse-scale computer, discussed throughout the later sections of this chapter, communication costs grow increasingly significant. Software designed without accounting for network latency and bandwidth limits, discussed earlier regarding cluster networking, can fail to achieve expected performance gains even on hardware with abundant raw parallel capacity.
Chapter Summary: Parallelism at Every Scale
This chapter traced parallelism across an enormous range of scales: hardware multithreading hiding latency within a single core, multicore shared memory multiprocessors coordinating through cache coherence, GPUs applying massive data parallelism to independent workloads, and clusters and warehouse-scale computers extending these same coordination principles to entire buildings of machines connected by carefully designed networks. At every scale, the same fundamental tension reappears: the potential performance benefit of dividing work must be weighed against the real cost of coordinating that divided work correctly.
Closing Thought: How This Full Series Fits Together
Across every article in this series, a single overarching theme has connected every topic: hardware achieves speed and capability not through any single dramatic breakthrough, but through the careful combination of simple, well-understood building blocks — instructions, arithmetic circuits, pipelines, caches, and parallel coordination mechanisms — each solving a specific, well-defined problem, and all working together to let human-written software run correctly and efficiently on physical silicon.