Shannon's Information Theory
Claude Shannon's 1948 paper 'A Mathematical Theory of Communication' is the foundational document of all digital communication.
The Most Important Paper in Communications
In 1948, Claude Elwood Shannon(1916–2001) published "A Mathematical Theory of Communication" in the Bell System Technical Journal. This single paper created the field of information theory, defined the concept ofinformation entropy, proved the existence of error-correcting codes, and established the channel capacity theorem — the theoretical maximum rate at which information can be transmitted reliably over any communication channel.
Before Shannon, communication was an engineering art — designers built systems and tested them empirically. After Shannon, communication became a science — designers could calculate the theoretical limits of any channel before building anything. Every modem, cellular system, satellite link, and fiber-optic cable in existence operates within the bounds that Shannon defined.
Information Entropy
Shannon defined information as a measurable quantity. Theentropy (H) of a source measures the average information content per symbol:
Where p(xᵢ) is the probability of symbol i. For a source withN equally likely symbols, H = log₂(N) bits/symbol. A fair coin flip has H = 1 bit/symbol. A fair die has H = log₂(6) ≈ 2.585 bits/symbol. An English text has H ≈ 1.0–1.5 bits/character (due to the statistical structure of language — some letters are more common than others).
The key insight: entropy defines the minimum number of bits needed to represent each symbol without loss. Entropy coding (Huffman coding, arithmetic coding) compresses data to near the entropy limit — this is the theoretical foundation of all data compression (ZIP, MP3, JPEG, H.264).
The Shannon-Hartley Theorem
Shannon's most famous result is the channel capacity theorem, which defines the maximum data rate (C) at which information can be transmitted reliably over a channel with bandwidth B and signal-to-noise ratio S/N:
Where C is channel capacity in bits/second, B is bandwidth in Hz, and S/N is the signal-to-noise ratio (linear, not dB). This equation says:
- Doubling the bandwidth doubles the capacity (linear relationship)
- Doubling the SNR increases capacity by only ~1 bit/s/Hz (logarithmic relationship)
- No amount of transmit power can exceed C — it is an absolute physical limit
- Any rate below C is achievable with arbitrarily low error probability — given sufficiently clever coding
Practical Examples
| Channel | Bandwidth | SNR | Shannon Limit | Actual Speed |
|---|---|---|---|---|
| POTS voice channel | 3.4 kHz | ~30 dB (1000:1) | 34 kbps | 33.6 kbps (V.34) |
| FM broadcast | 200 kHz | ~40 dB (10,000:1) | 2.66 Mbps | — (analog, not digital) |
| 802.11ac WiFi | 80 MHz | ~35 dB | ~533 Mbps/Hz | 433 Mbps (1SS) |
| Fiber optic (C-band) | ~4 THz | ~20 dB | ~27 Tbps | ~10 Tbps (lab record) |
| 5G mmWave | 400 MHz | ~15 dB | ~2 Gbps | ~1 Gbps (real-world) |
Error Correction: Approaching Shannon's Limit
Shannon proved that reliable communication is possible at any rate below capacity — provided sufficiently powerful error-correcting codes are used. But he did not say how to build them. For 50 years, engineers designed codes that worked but fell far short of the Shannon limit:
- Hamming codes (1950): Simple single-bit correction — far from capacity
- Reed-Solomon codes (1960): Burst error correction — used in CDs, DVDs, deep-space
- Convolutional codes + Viterbi decoding (1967): Near-optimal for convolutional codes — 3–5 dB from capacity
- Turbo codes (1993): Berrou, Glavieux, and Thitimajshima — within 0.7 dB of the Shannon limit using iterative decoding. This was a revolution — for the first time, practical codes approached the theoretical limit.
- LDPC codes (1996 re-discovery): Low-Density Parity-Check codes — originally invented by Gallager in 1960, rediscovered in the 1990s. Used in WiFi (802.11n/ac/ax), 5G NR, and DVB-S2.
- Polar codes (2009): Erdal Arıkan — the first codes mathematically proven to achieve Shannon capacity. Adopted as the control channel coding scheme in 5G NR.
Shannon's Legacy
Shannon's information theory underpins every digital communication system in existence:
- Data compression: ZIP, JPEG, MP3, H.264 — all use entropy coding near Shannon's entropy limit
- Error correction: Every digital link uses codes that approach Shannon capacity — turbo codes, LDPC, polar codes
- Channel capacity: Every modem, WiFi chip, and cellular system is designed to approach the Shannon limit for its channel
- Cryptography: One-time pads — the only cryptosystem proven to be information-theoretically secure (Shannon, 1949)
Shannon is sometimes called the "father of the digital age." His 1948 paper is the most cited paper in the history of electrical engineering, and his channel capacity theorem is the most important equation in communications — more fundamental than Ohm's law for anyone who designs TX/RX systems.