To allow information from several independent sources to be additive, Shannon gave a measure in terms of information bits. One "bit'' is the amount of information required to distinguish between two equally likely symbols, two bits are required to distinguish one symbol out of 4 and so on. In general, if there are M equally likely symbols, one needs log₂ M bits to represent them correctly.

A typical communication system consists of a transmitter, a channel and a receiver. A transmitter sends a signal over a communications channel to a receiver that collects the signal for further signal processing. The signal consists of a series of symbols, which convey some average amount of information, R, measured in bits per symbol. The uncertainty of the receiver before receiving symbols and after reception:

To find the maximum information from these equations, the symbols appearing at the receiver must be equally likely, (so that H_before = log₂  M  ) and every symbol must be exactly identified (no uncertainty left after reception, H_after = 0). Under these circumstances the information is R_maximum = log₂  M  . If there is any noise, or the symbols are not equally likely (H_before is less than H_equal) then this simple formula must not be used and R_maximum.

If the symbols are sent at a rate of W_s symbols per second, then the channel carries W_sR bits per second.

Shannon defined the "channel capacity" of a communications system and showed that it is:

Shannon proved a remarkable theorem about the channel capacity. One part of the theorem says that we cannot send information at a rate faster than the channel capacity. If we try to do this (i.e., W_sR > C), a quantity of noise will be received that limits the rate to C. The other part of the theorem is surprising: if we transmit at any rate less than or equal to the channel capacity ( W_sR is less than or equal to C), then the transmission is possible with as low an error rate as we may desire.

There is a price to be paid to get a low error rate: we must carefully encode the signal before transmission and then carefully decode it afterward. Although both steps require a delay, the overall transmission rate can approach C. Unfortunately the derivation and the proof of the theorem do not tell us how to make codes which allow transmission at rates close to C. Nevertheless, the formula is useful for understanding and designing communication systems, and methods have been found for creating "good'' codes.

Infomation Theory Links

• Information Theory Society (IEEE)
• A documentary on Claude Shannon
• yet2bdefined
• Information Theory, Inference, and Learning Algorithms by David MacKay
  (Available On-line, is a great book to read)