Since the Gaussian is the maximal (Shannon's) entropy distribution in unbounded real spaces, I was wondering whether the tendency of cummulative statistical processes with the same mean having a Gaussian as the limiting distribution can be in some way physically related with the increase of (Boltzmann's) entropy in thermodynamical processes.
In Johnson, O. (2004) Information Theory and The Central Limit Theorem, Imperial College Press, we can read:
"It is possible to view the CLT as an anlogue of the Second Law of Thermodynamics, in that convergence to the normal distribution will be seen as an entropy maximisation result"
Could anyone elaborate on such relationship and perhaps point to other non-obvious ones?
This is actually a deep philosophical question, and in your commentary you include what could be a philosophical flaw depending on what you mean by "physical": "can be in some way physically related ..."
Mathematical theorems are statements of logical coherence: "if this, then that"! They must be linked to reality, provided reality itself is coherent - a very large assumption, but one that all physicists make!
The relation of the CLT to thermodynamics is precisely through the "fact" that nature is coherent. In this sense Johnson made an error in his textbook in calling this relation an "analogue" - it is not an analogue but an instantiation: nature satisfies the requirements of the theorem and consequently embodies it.
So, what do you mean by "physically related"? If you are referring to one of the essential features of nature, viz. its coherence, that is itself the "relation"! Remember that etymologically, "physics" is "the knowledge of nature", where of course "nature" for the Greek was "everything that is" (and note, this is a pre-Christian conception: the Greeks believed that the gods were natural, that is, part of the natural world).
But I suspect that by "physically related" you have in mind some sort of mechanism. I think this is a mistake. The two ideas are related very deeply, actually at an ontological level.
If you're interested in the mathematical aspects of convergence to a `state of maximal entropy' in statistical theorems, there are some very recent (2013 or 2014) papers by S. G. Bobkov, G. P. Chistyakov and F. Götze which have appeared in top probability journals.
Also, I remember that a few years ago (after Oliver Johnson's book) it was proved that the entropy convergence to the maximum is indeed increasing (converging to the maximum does not by itself ensure that ever smaller oscillations are not present). A simple proof was published in an Information Theory journal but I can't remember the reference.
I try to clarify the problem by a simple example. suppose we impose a high temperature on a mid point of a metallic bar, for convenience suppose at initial time, t=0 the distribution of temperature at mid point is close to a Dirac delta distribution,by time elapse, this temperature distribution evolves to normal distribution that tends to be flatten after a long time, with increasing deviation, this example reveals that the entropy of system continued to be increased till the distribution approaches to a uniform distribution. consequently increasing entropy tend to make a uniform distribution rather a normal distribution, another example is releasing a small amount of perfume in a chamber, the final distribution rides toward a uniform distribution.Indeed without limiting conditions, a uniform distribution maximize the Shannon entropy S=-∑pi log pi .
@Chris: I'm not a physicist or a mathematician but I understand your concerns about the meaning of the term "physically related". When I wrote that, I wasn't putting too much thought on it. I think a precise meaning would depend on the school of thought I subscribe to when it comes to philosophy of mathematics itself (platonism, empiricism, logicism, formalism, etc. see: http://en.wikipedia.org/wiki/Philosophy_of_mathematics)
Nevertheless, my philosophical position is still somewhat fuzzy on this matter. Still, my post-hoc reflection is that by "physically related" I was asking whether probability theory is indeed an accurate and consistent model to conform to our intuition regarding the 2nd Law, what kind of reminds me of Prigogine's probabilistic objective approach of explaining unstable systems far from equilibrium.
For example, defining addition of random variables as a convolution between probability measures gives us a precise statement of which results we expect to be more likely to be observed by following the underlying logic of set and measure theories. But it usually makes no sense in thinking that random variables do exist as physical entities. It is only after one models the outcomes of finite trials by assigning a probability distribution to them that the logic is shown to hold approximately.
The point of "physically related" would then be that such probability models can be said to be consistent with physical reality because there may exist physical mechanisms that renders the assumptions and principles of both CTL and the 2nd Law logically equivalent.
For instance, because the uncertainty regarding the velocity of a gas molecule can be modeled in terms of a Gaussian distribution, its "kinetic energy" will be expressed in terms of the "variance" of that distribution.
Since the 2nd Law can be invoked to indentify the most probable macrostates and because "kinetic energy", along with "temperature" and "volume", partially describes such macrostates, then we are compeled to believe that the concept of "variance" of a Gaussian distribution may be a reasonable approximation to whatever mechanisms are responsible for governing the observed energy dispersion in such thermodynamical systems.
I don't know whether this makes sense, but, by this account, I'm still not grasping why asking for a physical mechanism that would render both CTL and the 2nd Law logically equivalent would be wrong.
I'd be glad if you could elaborate on that. Thanks!
Good question! A relationship between CTL and the 2nd Law is suggested in this paper here:
http://www.ams.org/journals/jams/2004-17-04/S0894-0347-04-00459-X/S0894-0347-04-00459-X.pdf
It says: "....one can now see clearly that convergence in the central
limit theorem is driven by an analogue of the second law of thermodynamics."
Don't you hate it when papers say they can "clearly" see something without any discussion at all? :-)
So this is just me making a guess at filling in the missing discussion: In the same way the 2nd Law governs (say) the expansion of a gas in a box and thermodynamic entropy increases, the above paper shows that in each step towards converging to the Central Limit the information entropy increases.
So for this to make sense to you, you need to:
(a) First believe in the analogy between information entropy and thermodynamic entropy.
(b) Understand that when you have thermal equilibrium, where the 2nd Law tells you that you cannot do net work, then the consequence of this is increased entropy.
The best mental picture I can give you to understand this is Feynman's ratchet & pawl. In non-equilibrium conditions the ratchet turns in one direction and does work. In equilibrium (when the 2nd Law says no net work is allowed) the ratchet fluctuates back and forth. To describe the motion of the wheel turning in one direction (ie. non-equilibrium case) requires precisely one bit of information and hence information entropy is low. To describe the fluctuations of the wheel (ie. equilibrium case) requires gazillions of bits of information and hence high information entropy.
This is my personal mental picture I have in my mind that helps me to believe in the analogy between informational and thermodynamic information. I describe some of these thoughts in detail in this paper here:
https://www.researchgate.net/publication/265965363
Article Allison mixtures: Where random digits obey thermodynamic principles
As far as I can recollect, CLT refers to the final outcome of a large number of independent processes, each of them providing a small impact on the system. Then, no matter what the detailed structure of each process is like, you get a Gaussian as distribution function for the output. Brownian motion is a textbook example, discussed e.g. by Chandrasekhar in his review paper in 1943 in Review of Modern Physics.
In contrast, maximisation of Boltzmann entropy in a macrostate of a N-body system follows from equiprobability of microstates in steady solutions of the Liouville equation for the N-body distribution function. (I will not mention the constraints on such maximisation, notably the conservation of energy).
Indeed, these things have something in common: 'equiprobability', i.e. the fact that the probability of a given microscopic configuration is exactly equal to the probability of any other microscopic configuration.
In CLT, equiprobability is just one of the hypotheses of the theorem.
In thermodynamics, equiprobability follows from the very structure of Liouville equation in steady state in both classical and quantum mechanics.
As for classical mechanics, Liouville equation in steady state reduces to the statement of vanishing Poisson bracket of the N-body Hamiltonian with the N-body distribution function. This is only possible if the distribution function depends on the total energy of the N-body system. Accordingly, the value of the distribution function is exactly the same for all microstates with the same total energy, and equiprobability of microstates follows.
As for quantum mechanics, you should replace the words 'Poisson bracket' and 'distribution function' with 'commutator' and 'density matrix' respectively; nothing else changes.
CLT and Second Law of Thermodynamics involve "large numbers", and therefore, they imply some type of convergence of towards an asymptotic mathematical behavior that it can be predicted with a great accuracy. However, I don't think that Second Law of Thermodynamics can be seem as a form of CLT. Of course, there will be many examples where such a connection is possible. For example, irreversible dynamical equations associated with Brownian motion, Langeving or Fokker Planck equations, can be related to CLT using random Gaussian noises. However, generality of this type of relationship cannot be overestimated.
Most of CLT's involve a linear combination of number N of independent random quantities (non-necessarily identically distributed), which leads to an asymptotic stable distribution (Gaussian, Cauchy, etc.). However, this type of mathematical arguments do not apply in many physical situations.
Magnetic dipoles, as example, are independent microscopic variables in a magnetic system with a very large number N of degrees of freedom that is put in thermal contact with a heat bath at constant temperature. When this magnetic system is near a critical point, magnetization per particle (average of dipole momenta) does not obey a Gaussian distribution because the microscopic random quantities (magnetic dipoles) are strongly correlated among them. Asymptotic distributions associated with CLT do not occur during the occurrence of a discontinuous phase transition. In these cases, two or more "macroscopic configurations" can coexist, that is, one can observe multi-modal distributions. Sometimes, such a multi-modal distributions can be approximated by a superposition of several Gaussian distributions. However, none of these mathematical anomalies involve any violation of second law of thermodynamics.
Second law of thermodynamics is intimately related to conection between thermodynamics and statistical mechanics. Any physical theory with a "probabilistic mathematical apparatus" will have an asymptotic limit of large numbers where predictions of this theory can be practically described by a "deterministic theory". As example, thermodynamics and classical mechanics are deterministic asymptotic theories for statistical mechanics and quantum mechanics, respectively. The first one arises when number of degrees of freedom of a system is very large, the second one, in the limit of high quantum numbers. These limit situations take place when their respective relevant functions, the entropy and the classical action, are much larger than their respective fundamental constants, Boltzmann (k) and Planck (h) constants.
The relationship between a probabilistic theory and deterministic theory is the content of a "correspondence principle". The correspondence of classical mechanics and quantum mechanics is the content of "Bohr's correspondence principle". The correspondence of thermodynamics and statistical mechanics is the content of "Einstein's postulate of classical fluctuation theory". By the way, usual textbooks on statistical mechanics never refer Einstein's postulate as a correspondence principle, but this is its true physical meaning.
Second Law of thermodynamics appears as the asymptotic deterministic behavior for some type of irreversible stochastic processes, which are somehow described by a set of real differential equations of first-order respect to time. Quantum mechanics, on the contrary, describes reversible stochastic processes that involves linear differential equations of first-order respect to time x the imaginary unit, so that, "oscillatory behaviors" are possible (waves). In many senses, quantum mechanics and statistical mechanics are very analogous. For example, these theories allow uncertainty relations that put precise limits to their respective deterministic theories. Perhaps, their most significant difference concern to the reversivle or irreversible character of their associated dynamical equations.
You can see more about this in the following paper:
Article Principles of classical statistical mechanics: A perspective...
By the way, "irreversibility" is always present is any physical theory with a statistical nature. Every physical theory is based on experiment. For a statistical theory, as quantum mechanics and classical mechanics, any experiment must involve "a preparation" and "a test". Sucesive repetition of preparations and tests allows to characterize the possible results using "probabilities". The main goal of such a statistical physical theory is to predict the results (the ocurrence probability) of the experiment starting from a given preparation. Naturally, this procedure involves a preferential direction in time. In a certain sense, this is the origin of irreversibility.
The initial preparations in statistical mechanics is what is referred to as "an statistical ensemble". While quantum mechanics involves uncountable ways to prepare quantum systems, statistical mechanics discussed in most of textbooks exhibits a hallmarked preference of a very reduced set of "a preparations", the well-known statistical microcanonical, canonical, gran canonical ensembles. Needless to say they never refer statistical ensembles as "preparations".
The connection is by entropy maximisation. Distributions with finite
second moment lead by convolution to a gaussian
distribution (basin of attraction, CLT) and maximisation
of entropy with constraint on the second moment
leads to a gaussian distribution (as opposed to maximisation with only normalisation constraint leading to a uniform distribution)
The convergence of a random variable to any distribution can be interpreted as maximization of entropy under appropriate constraints.
Entropy: A concept that is not a physical quantity
https://www.researchgate.net/publication/230554936_Entropy_A_concept_that_is_not_a_physical_quantity
- Badges
- Science topic