Information Theory and Thermodynamics, A parallelism

The word which strongly connects Thermodynamics and Information Theory is Entropy. In Information theory, the Gaussian distribution (figure on the right side) is the distribution with the maximum entropy which is defined as 'S' below.

This also means that minimum amount of information (mean and variance) is sufficient to describe the distribution completely without knowing any further information. Addition of any further information to the system will only reduce the entropy. Similarly, in thermodynamics as the system tends to equilibrium, the amount of information required to describe the system reduces and sufficient to know the average properties of the system.

In a process, as the number of contribution to noise(with any kind of distribution) increases, the overall noise tends to Gaussian distribution, which is a consequence of central limit theorem. This physically means that as more and more noise is added to a system, the amount of information with which one can characterize the noise reduces and the description has to be done with a minimum assumption on the noise. A thermodynamics counterpart of number of noise would be the number of molecules with which one will have to characterize the system. As the number of molecules increases, the amount of information which one has to keep track of increases and consequently one will be left with no choice than to work with macroscopic properties which allows to describe the system with least information.

One more interesting analogy is central limit theorem and second law of thermodynamics. central limit theorem says addition of noises makes the overall noise tend to gaussian, that is move to a maximum entropy. Second law of thermodynamics also says that any process moves towards maximum entropy.

One of the classic papers to read about this is 'Information theory and Statistical Mechanics' by E.T. Jaynes (physical Review, vol 106, pg. 320).

A basic understanding of Central limit theorem can be got from wikipedia!

I would like to thank prof. Shankar Narasimhan whose lectures on 'Multivariate data analysis' was a preeminent factor in my attempt to write this blog.