Statistical communication theory concerns itself with the mathematical concept of information. Cherry is correct that this does not include the more vague, everyday uses of the word information. Instead, the study of information concerns itself only with signals, where each signal prompts a selection or discrimination. As Cherry points out, if there is no doubt or need to make a selection between alternatives, there is no information being transmitted.
However, information alone does not equal human communication. As Cherry states, it is assumed that as long as the signals are transmitted correctly, the resulting messages "will have 'meaning', value, truth, reliability, timeliness, and all their other properties" (p. 168). Even assuming this is true, a statistical or informational theory of communication gives no clue as to the nature of these message properties or how they arise from the transmitted signals.
Since an informational theory of communication concerns itself only with signals, which have some physical manifestation, it can be more objective and directly empirical than other communication theories. After transmitting a great number of signal samples, information theorists can then statistically analyze the frequency, variation, and other of physical aspects of the signals to discover patterns and limits.
Information rate = n log2 N.
Hawaiian language = 1 log2 16 = 4 bits per second.
English language = 1 log2 32 = 5 bits per second.
Binary = 1 log 2 2 = 1 bit per second.
This assumes that only one sign (letter) is being transmitted per second; usually n is much higher. Also, it assumes a random distribution of signs; that is, that each sign is equally probable to occur. While this may be true in binary code, it is rare in natural languages, such as English, where e is much more common than z, and so carries less information.
Time averaging would be used on a stationary source--that is, a source that produces information statistics that are relatively unchanging over a long period of time. Because of this, a series of samples can be taken from this single source and then averaged. An example of a stationary source might be yearly rainfall at a certain location.
A moving source--a source that produces statistics that exhibit a macroscopic trend--can be averaged using ensemble averaging. With this technique, a number of macroscopically identical sources are samples at the same time, and averages of their microscopic statistics are computed for all the sources for one time sample at a time, rather than over the time. An example of a moving source might be daily temperatures. Taking samples over a single year would be less useful than taking an average for each date over a number of years, which would allow you to say this October 1st is particularly cold, hot, or average.
The Capacity Theorem states the maximum rate information that can be transmitted through a noisy channel without error. It is a largely theoretical limit, since it gives a mathematical limit--up to that limit, it is possible to transmit with an arbitrarily few number of errors. In practice, a channel still produces some errors. If engineers believe the Capacity Theorem, they need not try to exceed this limit, because they can never do it. It is like trying to travel faster than light if you believe the General Theory of Relativity.
Shannon's "ideal coding" is efficient but requires long time delays. He would pack the most information into each sign (or "letter"), which could best be done with binary data, where each sign carries information independently. On the other hand, Mandelbrot assumes that information comes in words, separated by the "space" sign. Though less efficient, this encoding is more practical when having a human conversation and using words. Words that occur most frequently tend to have lower costs (be shorter and quicker to say).
Entropy is the mathematical opposite of information. If information is the reduction of choices and doubt, entropy is the increase in disorder and chaos. The Second Law of Thermodynamics states that any closed system will tend toward increased entropy. This can give a sense of time--as time moves forwards, systems run down.
The "old problem of Maxwell's demon" is a scenario in which a demon is receiving information about the particle motions of a gas, and using that information to set up a perpetual motion machine. Though receiving information, the demon does not seem to be increasing the entropy of the system. However, the demon is actually part of the system, and so it is supposed that his actions do increase the entropy of the system as a whole--more so than he can reduce it by harnessing information.
Entropy, as the opposite of the mathematical definition of information, plays a role in a statistical theory of communication. Essentially, entropy is the doubt removed by the transmission of information.
Cherry, Colin. On human communication: a review, a survey, and a criticism. Cambridge: Technology Press of Massachusetts Institute of Technology, 1957.
CIS: Week 6
|Last Edited: 01 Oct 2004|
©2004 by Z. Tomaszewski.