Complex Systems in Neurobiology Language and Speech

Sp&Hrg 803
Complex Systems in Neurobiology, Language, and Speech

Day 3
19 February 1998
Information 1

Summary of this Day's Class

We spent the first part of the class discussing the articles.
This was kept short and we prepared ourselves for Information Theory by reviewing Logarithms. I have included the transparencies below in Word 97, Word 6.0 and pdf formats:
LogRefresher.pdf - PDF Format, 16 kb.
LogRefresher.doc - Word 97 Format, 9 kb.
LogRefresher-6.doc - Word 6.0 Format, 24 kb.

After reviewing logarithms we began an introduction to Information Theory where we were introduced to the idea of surprise, and then the Shannon Information which is the expectation value of the surprise (or loosely speaking - the average surprise). We briefly looked at the relationship between information and entropy.
Download the transparencies for Information Theory Part 1 here:
InfoTheory1.pdf - PDF Format, 12 kb.
InfoTheory1.doc - Word 97 Format, 51 kb.
InfoTheory1-6.pdf - Word 6.0 Format, 147 kb.

Homework Assignment 3 - Due 26 Feb 98

1. Consider a coin and a six-sided die.

How many states can the flipped coin have (list them)?
How many states can the die have (list them)?
How many states can the flipped coin plus the thrown die have together (list them)?
Verify that the number of states of the coin and the die together is the number of states of the coin times the number of states of the die.
Verify that the logarithm of the number of states of the coin plus the logarithm of the number of states of the die equals the logarithm of the number of states of the die and the coin together.
Do this using the Log base 10.
Try it also with the natural logarithm (Log base e).
Can you show it with Log base 2?

2. Calculate the entropy for the following coins (with different probabilities for heads and tails). Remember that the P(T) = 1 - P(H) so its easy to find P(T) if you know P(H).: a. P(H) = 0.5; c. P(H) = 0.4; d. P(H) = 0.3; e. P(H) = 0.1; f. P(H) = 0.6 (Notice there is a symmetry interchanging H for T); Graph your results.; Extra: Can you solve it completely for all possible P(H) and graph the entropy vs. P(H)?; (No extra credit will be give for the extra parts. They are just a little more challenging.)

3. Calculate the entropy for a system with four states using Log Base 2.: P(a) = 1/2, P(b) = 1/4, P(c) = 1/8, P(d) = 1/8; Can you see the relation between the entropy and the average number of yes-no guesses you would have to make to determine the state?; Extra: If you could change the probabilities what would give you the maximum entropy and what would its value be?