Claude Shannon - A Mathematical Theory of Communication (1948)

History / Edit / PDF / EPUB / BIB /
Created: August 13, 2015 / Updated: July 24, 2025 / Status: in progress / 2 min read (~206 words)
artificial general intelligence

A Mathematical Theory of Communication is a critical piece of knowledge for anyone interested in the topic of information.

Capacity is defined as

$$ C = \lim_{T\to\infty} \frac{\log N(T)}{T}$$

We want the capacity of a discrete channel to be as high as possible (highest throughput possible).
At the same time, we want the message that are transmitted to be as small as possible.

Binary entropy function

$$ H = -(p \log p + q \log q)$$

  • A transducer can be described by two functions:

    $$ y_n = f(x_n, \alpha_n)$$

    $$ \alpha_{n+1} = g(x_n, \alpha_n)$$

    where

  • $x_n$ is the $n^{th}$ input symbol
  • $\alpha_n$ is the state of the transducer when the $n^{th}$ input symbol is introduced
  • $y_n$ is the output symbol (or sequence of output symbols) produced when $x_n$ is introduced if the state is $\alpha_n$