There are many many statements in the literature which say that information is the same as entropy. The reason for this was told by Tribus. The story goes that Shannon didn't know what to call his measure so he asked von Neumann, who said `You should call it entropy ... [since] ... no one knows what entropy really is, so in a debate you will always have the advantage' (Tribus1971).
Shannon called his measure not only the entropy but also the "uncertainty". I prefer this term because it does not have physical units associated with it. If you correlate information with uncertainty, then you get into deep trouble. Suppose that:
but
since they have almost identical formulae:
How could that be? Information is the very opposite of randomness!
The confusion comes from neglecting to do a subtraction:
If you use this definition, it will clarify all the confusion in the literature.
Note: Shannon understood this
distinction and called the uncertainty which is subtracted the 'equivocation'.
Shannon (1948) said on page 20:
The mistake is almost always made by people who are not actually trying to use the measure. As a practical example, consider the sequence logos. Further discussion on this topic is in the http://www.ccrnp.ncifcrf.gov/~toms/bionet.info-theory.faq.html under the topic I'm Confused: How Could Information Equal Entropy?
For a more mathematical approach, see the Information Theory Primer.
Some questions and answers might make these isues more clear.
@article{Tribus1971, author = "M. Tribus and E. C. McIrvine", title = "Energy and Information", journal = "Sci. Am.", volume = "225", note = "{(Note: the table of contents in this volume incorrectly lists this as volume {\bf 224})}", number = "3", pages = "179-188", month = "September", year = "1971"}
@article{Machta1999, author = "J. Machta", title = "{Entropy, Information, and Computation}", journal = "Am. J. Phys.", volume = "67", pages = "1074-1077", year = "1999"}"The results of random processes usually have high information content". "Randomness and information are formally the same thing." He also shows an equation relating "Shannon information" to the uncertainty function. This is a perfect example of total confusion on this issue!
@article{Padian2002, author = "K. Padian", title = "{EVOLUTION AND CREATIONISM: Waiting for the Watchmaker}", journal = "Science", volume = "295", pages = "2373-2374", year = "2002"}"In information theory, the term can imply increasing predictability or increasing entropy, depending on the context." Kevin Padian, who wrote the review, reports that the error came from the book he was reviewing:
Intelligent Design Creationism and Its Critics Philosophical, Theological, and Scientific Perspectives Robert T. Pennock, Ed. MIT Press, Cambridge, MA, 2001. 825 pp. $110, �.95. ISBN 0-262-16204-0. Paper, $45, �.95. ISBN 0-262-66124-1.
@article{Allahverdyan.Nieuwenhuizen2001, author = "A. E. Allahverdyan and T. H. Nieuwenhuizen", title = "{Breakdown of the Landauer bound for information erasure in the quantum regime}", journal = "Phys. Rev. E", volume = "64", pages = "056117-1--056117-9", year = "2001"}This is an example of the typical physicists' muddle about "erasure" in which they set the state of a device to one of several states and call this a "loss of information". But setting a device to one state (no matter what it is) decreases the entropy and increases the information. The main mistake that the physicists make is not having any real working examples. It's entirely theoretical for them. (These people believe that they can beat the Second Law. I would simply ask them to build the perpetual motion machine and run the world power grid from it before making such a claim.)
@article{Crow2001, author = "J. F. Crow", title = "{Shannon's brief foray into genetics}", journal = "Genetics", volume = "159", pages = "915--917", year = "2001"}He confounds information with uncertainty, but forgot the minus sign on the sum p log p formula. He also confounded information with entropy. Finally, he claimed that "a noisy system can send an undistorted signal provided that the appropriate error corrections or redundancy are built in". This is incorrect since there will always be error, but Shannon's channel capacity theorem shows that the error can be made as low as desired (but not zero as this author claims).
"Entropy measures lack of information; it also measures information. These two conceptions are complementary. " The meanings of entropy, Jean-Bernard Brissaud, Entropy 2005, 7[1], 68-96.
2006 Oct 19: Martin Van Staveren pointed out that
at the top of page 22 Shannon's 1948 paper, it seems to be suggested, that part of the received information is due to noise. This is obviously a slip of the pen of Shannon, as he merely tries to explain, in words, that the information rate R is the initial uncertainty minus the uncertainty due to the noise; but he calls H "information" instead of "entropy".He further pointed out that much of the confusion may have come from Weaver:
see this: http://www.uoregon.edu/~felsing/virtual_asia/info.html
This is part of the intro that Weaver wrote for "The mathematical theory of information". Some people even refer to "Shannon-Weaver theory" because of this intro.
section 2.5 of this intro: noise generates "spurious", or "undesirable" information, whatever that may mean. The section also introduces the esoteric notion of "meaningless information", contrary to what Shannon himself says in the body of the text. I think that Weaver's arrogance in thinking that he had to "explain" Shannon, has done a big disservice to information theory, which really is only probability theory.
"A Bit Confused. Creationism and Information Theory" Skeptical Inquirer, 3/1/01 David Roche. (link 1) (link 2). He made a mistake in his description of Shannon information. What he described is the Shannon uncertainty. Shannon information is a difference between uncertainties. Because of his confusion, he associates disorder with information.
2009 Jan 21: 6.050J Information and Entropy (Spring 2008) an MIT Open Courseware course. The preface reads: "Only recently has entropy been widely accepted as a form of information." which is, of course backwards.
Also, the statement "Second Law states that entropy never decreases as time goes on" is wrong since the entropy of a system can decreas if heat leaves the system - that's how snowflakes form!
At least they admit: "In fact, we may not yet have it right."!!
Schneider Lab
origin: 1997 January 4
updated:
2009 Jan 21: MIT OCW