Information Theory

The Master Story

• Source coding compresses to the entropy limit
• Channel coding protects against noise
• Information theory asks: What is fundamentally possible?
• The theoretical crown of the Communication Universe

Self-Information

• Information = "Surprise" or Unexpectedness
• I(x) = -log₂ p(x) (measured in bits)
• A rare event (p ↓) carries massive information
• A certain event (p = 1) carries zero information

Shannon Entropy

• H(X) = -Σ p(x) log₂ p(x)
• The average information per source symbol
• Measures the fundamental uncertainty of the source
• Maximum entropy (H_max = log₂ M) occurs when all symbols are equally likely

Binary Entropy

• H(p) = -p log₂ p - (1-p) log₂(1-p)
• Curve peaks at exactly 1 bit when p = 0.5 (fair coin flip)
• Approaches 0 as predictability increases (p → 0 or p → 1)
• Forms the compression limit for binary sources

Mutual Information

• I(X;Y) = H(X) - H(X|Y)
• How much observing Y reduces uncertainty about X
• Measures what actually "survives" the channel
• Entropy = uncertainty. Mutual Info = uncertainty removed.

The Noisy Channel

• Described mathematically by transition probability P(y|x)
• Binary Symmetric Channel (BSC): Flips bits with probability p
• Binary Erasure Channel (BEC): Loses bits (returns "?")
• Abstracts physical hardware into pure mathematical models

Channel Capacity

• C = The absolute maximum reliable information rate
• Mathematically: C = max I(X;Y) over all input distributions
• The boundary line of physics and information
• Different inputs produce different mutual information

Shannon-Hartley Theorem

• C = B log₂(1 + S/N) (for band-limited AWGN channels)
• The most famous equation in communication engineering
• C = Capacity in bits/second
• B = Bandwidth (Hz), S/N = Linear Signal-to-Noise Ratio

Bandwidth vs SNR

• Bandwidth (B) multiplies the capacity linearly
• SNR sits inside the logarithm (diminishing returns)
• Doubling Bandwidth ≠ Doubling SNR
• You can trade excess BW to compensate for terrible SNR

Spectral Efficiency

• η = R / B (Information rate per Hz of bandwidth)
• Units: bits/s/Hz
• Bounded theoretically: η ≤ C / B = log₂(1 + S/N)
• Higher-order modulations (e.g., 256-QAM) push this limit

The Shannon Limit

• If Rate (R) < C: Arbitrarily reliable communication is possible
• If Rate (R) > C: Reliable communication is IMPOSSIBLE
• This is an asymptotic, physical law of nature
• No clever coding trick can bypass it

Operational Regimes

• Power-Limited: Deep space. Huge bandwidth, tiny signal. Capacity relies on C ≈ B(S/N)/ln(2).
• Bandwidth-Limited: Cellular. Scarce spectrum, good signal. Capacity relies on C ≈ B log₂(S/N).

Eb/N0 Connection

• E_b = Energy per information bit. N_0 = Noise density.
• Replaces raw SNR to fairly compare digital coding schemes
• The famous waterfall BER curve crashes into the Shannon limit wall (approx -1.59 dB for infinite bandwidth)

Typical Sets (AEP)

• Asymptotic Equipartition Property
• Most probability mass lies in a tiny subset of "Typical Sequences"
• Size of typical set ≈ 2^(nH)
• Explains why entropy H is the true compression floor

The Grand Sequence

• Shannon's Separation Theorem allows us to decouple:
• Realm 11: Compress unnecessary redundancy out
• Realm 12: Protect data with structured redundancy
• Realm 14: Push the aggregate rate up against Capacity C

16 — The Capacity Laboratory

This is the theoretical peak of the Communication Universe. You now understand what information is, how to strip it to its essence, how to armor it against the void, and the absolute physical limits of how fast it can travel. You have moved from constructing circuits to mastering the fundamental laws of nature.