Why Comparing a CPU and a GPU Is Not Straightforward
The CPU and GPU architectures discussed earlier in this series are optimized for fundamentally different workloads, which makes a simple side-by-side comparison of raw computational speed misleading. A fair comparison needs a model that accounts for both how fast each can compute and how fast each can move data, since real workloads are often limited by one or the other.
The Roofline Model
The Roofline Model provides exactly this kind of balanced comparison, plotting achievable performance against a workload's Arithmetic Intensity, the ratio of computational operations performed per byte of data moved from memory.
Two performance ceilings:
1. Peak computational throughput
(limited by available ALUs and clock speed)
2. Peak memory bandwidth
(limited by how fast data can be moved from memory)
A workload's actual achievable performance is
limited by whichever ceiling it hits first,
based on its specific arithmetic intensityA workload with low arithmetic intensity, performing few calculations per byte of data moved, is typically limited by memory bandwidth, discussed earlier in this series regarding the memory hierarchy, no matter how many ALUs are available. A workload with high arithmetic intensity can potentially achieve closer to peak computational throughput, since it is not as constrained by data movement.
Comparing the Intel Core i7 960 and NVIDIA Tesla GPU
Applying the roofline model to real hardware, such as the Intel Core i7 960 and an NVIDIA Tesla GPU, reveals that neither processor is universally superior. The GPU's massive number of simple cores, discussed earlier in this series, gives it a much higher peak computational throughput ceiling, making it the better choice for workloads with high arithmetic intensity and abundant data parallelism, while the CPU's more sophisticated per-core design, including out-of-order execution and branch prediction discussed earlier in this series, makes it better suited to workloads with lower arithmetic intensity or significant branching and sequential dependencies.
Applying These Concepts: Matrix Multiply Across Multiple Processors
Matrix multiplication, discussed at multiple points throughout this series regarding subword parallelism, instruction-level parallelism, and cache blocking, can be extended one final step by distributing different portions of the matrices across multiple independent processors in a shared memory multiprocessor, discussed earlier in this series.
Sequential: one processor computes
all rows of the result matrix
Parallel across processors: each processor
computes a distinct subset of rows,
all working simultaneously on the shared matricesBecause the individual row computations are independent of each other, this workload scales well across multiple processors, though the actual achieved speedup, following Amdahl's Law and the real measurement factors discussed earlier in this series, will be somewhat less than the theoretical linear ideal due to memory bandwidth contention between processors accessing the shared matrix data.
Why This Combination of Techniques Matters
The fastest practical implementations of matrix multiplication combine nearly every technique covered throughout this entire series: subword parallelism within a single instruction, instruction-level parallelism across multiple execution units, cache blocking to respect the memory hierarchy, and now multiprocessor parallelism across many independent cores or machines, demonstrating how the layered concepts from every chapter of this book work together in a single, practically important computation.