Ontological Layer: Concurrent Computation

Parallel Computing

Many Machines of Thought Working as One

The Multiplicity of Execution

When a single pipeline reaches its physical limits, the only way forward is multiplicity. Realm 10 expands computation from one processor to many. We categorize these architectures via Flynn's Taxonomy (SISD vs SIMD vs MIMD) and examine how Multicore systems share memory. We confront the immense complexities of Cache Coherence, Synchronization, and Deadlock, and calculate the mathematical limits of speedup governed by Amdahl's Law.

SINGLE CORE → MULTICORE (MIMD) vs GPU (SIMD) → SHARED MEMORY → SYNCHRONIZATION (LOCKS) → CACHE COHERENCE (MESI) → AMDAHL'S SPEEDUP LIMIT

The 12 Parallel Nodes

Ontological Mapping: From Concurrency to Clusters

CHAMBER 01

The Idea of Parallelism

Multiple operations occurring simultaneously
Work decomposition into sub-tasks
Reducing overall execution wall-time
Throughput vs Latency optimization

CHAMBER 02

Concurrency vs Parallelism

Concurrency: Dealing with multiple things at once (Interleaving)
Parallelism: Doing multiple things at once (Hardware)
Bit-level, Instruction-level, Thread-level
Task-level vs Data-level parallelism

CHAMBER 03

Flynn's Taxonomy

Instruction Streams vs Data Streams
SISD: Classic sequential Von Neumann
SIMD: Vector processors, GPU foundation
MIMD: Modern Multicore and Clusters

CHAMBER 04

SIMD & Vector

Single instruction applied to multiple data points
Wide registers (e.g., 256-bit AVX)
Data-level parallelism
Highly efficient for graphics and matrices

CHAMBER 05

Multicore Processors

Multiple processing units on one die
Private L1/L2 caches per core
Hardware threads vs Software threads
Homogeneous vs Heterogeneous (big.LITTLE)

CHAMBER 06

Shared-Memory

All processors access the same address space
Communication via shared variables
Implicit communication, explicit synchronization
Symmetric Multiprocessing (SMP)

CHAMBER 07

Distributed-Memory

Each node has private, inaccessible memory
Communication via explicit Message Passing
Highly scalable (Clusters/Supercomputers)
No hardware cache coherence required

CHAMBER 08

Cache Coherence

Ensuring multiple caches hold consistent data
Write Invalidation vs Write Update
Snooping bus protocols
MESI Protocol (Modified, Exclusive, Shared, Invalid)

CHAMBER 09

Synchronization

Protecting Critical Sections
Locks, Mutexes, and Barriers
Race Conditions (Timing dependencies)
Deadlock (Mutual infinite waiting)

CHAMBER 10

Amdahl's Law

Speedup = 1 / ((1 - P) + P/N)
The Serial Fraction determines maximum theoretical speedup
Diminishing returns of adding processors
Gustafson's Law (Weak Scaling alternative)

CHAMBER 11

CPU vs GPU

CPU: Few heavy cores, optimized for sequential speed & branching
GPU: Thousands of light cores, optimized for massive data throughput
NUMA (Non-Uniform Memory Access)
Heterogeneous Accelerators

FINAL PROTOCOL

12 — Concurrency Laboratory

1. SERIAL: 25%, PARALLEL: 75%

2. N = 4 CORES

3. SPEEDUP = 1 / (0.25 + 0.75/4) = 2.28x

[1] CALCULATE AMDAHL'S LIMIT [2] TRACE MESI STATES [3] IDENTIFY FALSE SHARING [4] RESOLVE DEADLOCK [5] MAP GPU THREADS

Parallel computing allows hardware to conquer problems that a single processor never could, but it introduces the devastating complexities of synchronization and coherence. With the physical hardware capabilities now completely mapped out, we must finally introduce the software entity that manages it all. This prepares us for Realm 11: The Operating System.

PARALLEL DIAGNOSTIC RUNNING