Summary of this Day's Class

We didn't finish everything from last time so we continued with Information Theory.
First we played a game which allows one to estimate the Shannon Information for the English language.
This exercised is derived from the fact that the Information (when calculated in base 2) is the average number of yes/no questions necessary to determine the state of a system. We used a modification of this idea in our exploration where we calculated the infromation and redundancy of the English language.
The rest of the class was spent looking at the way that the structure in language decreases the Shannon Information. This is accomplished by both a deviation from equiprobability of the symbols and correlations among symbols in a symbol string such as a sentence. By decreasing the information by adding structure we reduce the possible number of messages that can be sent, however we also reduce the possibility of error (since error is defined by the incorporation of structure).
Here we have two constraints that must be simultaneously optimized. This leads to the complexity of language.

I tried to find some books on Information Theory:

Gatlin's book is perhaps a good starter, but one must be careful not to take everything too seriously when it comes to genetics since it is quite old. The ideas are there, but be cautious:
Gatlin, Lila L. 1972. Information Theory and the Living System, Columbia University Press, NY.


Herdan's book applied information theory to linguistics. Although interesting, it has a lot of math:
Herdan, Gustav. 1962. The Calculus of Linguistic Observations, Mouton & Co., 'S-Gravenhage.


A Santa Fe Institute Proceedings has many papers on the topic of information theory, entropy, and complex systems:
Zurek, Wojchiech H. (Ed.) 1990. Complexity, Entropy and the Physics of Information, Addison Wesley Publishing Co., Reading MA.


An excellent textbook in information theory is:
Cover, Thomas M. and Thomas, Joy A. 1991. Elements in Information Theory, John Wiley & Sons Inc., NY.

Homework Assignment 4 - Due 5 March 98

1. Let's pick a number a=2

Now let's start with a number x0 = 1/4 = 0.25

Use your calculator and calculate a x0 (1-x0).

The first time you should get 2*0.25*(1-0.25) = 0.375

Now call this answer x1 and find x2 using the same rule above.

Do this again and again until your get an answer that converges.

What is this number converging to?

The formula above is:

x(n+1) = a xn (1-xn)

where n and n+1 are indices describing the nth time going through the equation

Now try this again for a = 3.1.

This time watch the numbers closely.

They won't converge to a single number, but will do something tricky.

What numbers are they converging to?

If you are excited by this you can try this again for a = 3.5.

What happens now?

If you are used to writing a computer program, let the computer iterate this for you for all sorts of values of a. The trick is to compute say 200 values of x and then plot the second 100 discarding the first 100.

Reading Assignment - Read the articles by 5 March 98

Deneuborg, J.J. and Goss S. 1989. Collective patterns and decision making, Ethology Ecology and Evolution, 1:292-311.

Gatlin, L.L. 1972. Information Theory and the Living System. Columbia University Press, NY. Chapter 5, pp. 107-116.

Schroeder, M. 1991. Fractals, Chaos, Power Laws: Minutes from an Infinite Paradise, W.H. Freeman and Co. pp: 1-25.