Why This Final Chapter Matters
Every mechanism covered so far in this series — instructions, arithmetic, pipelining, and the memory hierarchy — has focused on making a single processor core run individual programs as fast as possible. This chapter shifts focus to systems built from many processors working together, and the unique software and hardware challenges that arise specifically from that shift.
Why Writing Parallel Programs Is Genuinely Difficult
Splitting a program's work across multiple cores sounds simple in principle, but in practice introduces problems that sequential programming never faces. A programmer must divide work into independent pieces, coordinate communication and synchronization between those pieces using mechanisms like the atomic instructions discussed earlier in this series, and manage shared data correctly across cores relying on the cache coherence guarantees also discussed earlier.
The Law That Limits Parallel Speedup
Even with perfect coordination, a fundamental limit remains: any portion of a program that cannot be parallelized still executes sequentially, bounding the maximum possible speedup no matter how many cores are added. This relationship, known as Amdahl's Law, explains why simply adding more cores to a system does not guarantee proportional performance improvement, especially for programs with a significant sequential portion.
If 90% of a program can be parallelized
and 10% must remain sequential,
even an infinite number of cores
can speed up the program by at most 10xClassifying Parallel Hardware: Flynn's Taxonomy
A classic framework, known as Flynn's Taxonomy, classifies computer systems based on how many simultaneous streams of instructions and data they process.
SISD (Single Instruction, Single Data): a traditional single-core processor executing one instruction on one piece of data at a time, the model implicitly assumed throughout most of this series before this chapter.SIMD (Single Instruction, Multiple Data): one instruction operates on multiple data elements simultaneously, the same concept introduced earlier in this series regarding subword parallelism.MISD (Multiple Instruction, Single Data): multiple instructions operate on the same data stream simultaneously, a rare configuration used mainly in specialized fault-tolerant systems.MIMD (Multiple Instruction, Multiple Data): multiple independent processors each execute their own instructions on their own data, describing most modern multicore and multiprocessor systems.
SPMD: A Practical Programming Model Within MIMD
Most real-world parallel programs use a specific style called SPMD (Single Program, Multiple Data), where every processor runs the same program, but operates on a different portion of the overall data, and can take different execution paths through that same program based on the specific data it is processing. This is technically a form of MIMD hardware usage, since each processor genuinely executes its own instruction stream, but with the practical simplicity of writing just one program rather than many completely different ones.
Vector Processing: A Structured Form of Data Parallelism
Vector Processors extend the SIMD concept further, operating on entire arrays, or vectors, of data using specialized instructions designed specifically for this purpose, historically important in scientific computing and closely related to the wide SIMD extensions like AVX discussed earlier in this series.
Why This Classification Framework Still Matters
Even though real modern systems often combine several of these categories simultaneously — a multicore MIMD system where each individual core also supports SIMD instructions — Flynn's Taxonomy remains a useful conceptual starting point for understanding the fundamental tradeoffs each parallel hardware style makes between programming complexity and achievable performance.