It is worth recalling a remark that Poincaré attributed to his friend Gabriel Lippmann, concerning the “law of errors” (“loi des erreurs”):

"Elle ne s'obtient pas par des déductions rigoureuses; plus d'une démonstration qu'on a voulu en donner est grossière, entre autres celle qui s'appuie sur l'affirmation que la probabilité des écarts est proportionnelle aux écarts. Tout le monde y croit cependant, me disait un jour M. Lippmann, car les expérimentateurs s'imaginent que c'est un théorème de mathématiques, et les mathématiciens que c'est un fait expérimental.”  (Poincaré, 1912)

[“It is not obtained by rigorous deduction; more of a rough demonstration of what we wished, among other things that on the assertion that the probability of deviations is proportional to the deviations. Everyone believes it though, M. Lippmann was saying to me one day, because the experimenters think that it’s a mathematical theorem, and the mathematicians that it’s an experimental fact.”]

This "law of errors is inexorably bound to the normal distribution. Furthermore, under most Bayesian interpretations of probability distributions aren't idealized outcomes of infinitely many "experiments", and as such the "physical interpretation" is just that: physical (and subjective).

Finally, note that there is one probability distribution (or one class/type/kind of such distributions) that differs in its manner of construction from virtually all others: probability in quantum mechanics (nobody outside of quantum physics relies on amplitudes for probabilities). Here, probability distributions are descriptions of physical systems and thus necessary have physical interpretations, but nobody agrees on what these are.

Asking about the physical interpretation of probability distributions requires (or would seem to) some physical interpretation of probability (after all, probability distributions are meaningless without a rigorous definition of probability), yet fundamental disagreements over the nature of probability have been the subject of debate in mathematics, physics, philosophy, and more for many centuries.

Dear Babak,

I can advice you to look at the following official documents with distribution function mentioned (for Natural Sciences):

1) WMO-No. 100 (Guide to Climatological Practices) published in 2011. The PDF file is attached. You should look at its chapters 5.3, 5.4 (pages 77-78).

2) WMO-No. 702 (Guide to Wave Analysis and Forecasting) published in 1998. The PDF file is attached. You should look at its chapter 9.4.1.2 (page 106).

Some brief answers about distribution function are given in my message by the following blocks of text.

9.4.1.2 The choice of distribution

The choice of distribution to fit all the data is open,

providing that F(0) = 0. There is no theoretical justification

for any particular distribution. This is an enormous

weakness in the method, particularly since considerable

extrapolation is involved. In practice, various distributions,

from among those which have been used with

some success over the years, are tried and the one giving

the best visual fit is accepted. These distributions, which

are defined in Annex III, are listed with comments in

Table 9.1.

The choice of distribution to fit the observed

maxima is limited by the theory of extreme values (see,

for example, Fisher and Tippett, 1928; Gumbel, 1958;

and Galambos, 1978) which shows that the distribution

of maxima of m values are asymptotic with increasing

m to one of three forms (FT-I, II and III)*. These can

all be expressed in one three-parameter distribution: the

generalized extreme value distribution (Jenkinson,

1955). This theoretical result is a great help in the analysis

of environmental data, such as winds and waves

for which the distributions are not known, and explains

the frequent application of extreme value analysis when

sufficient data are available. As usual, the theory

contains some assumptions and restrictions, but in

practice they do not appear to impose any serious

limitations. The assumption that the data are identically

distributed is invalid if there is an annual cycle. Carter

and Challenor (1981(a)) suggest reducing the cycle’s

effect by analysing monthly maxima separately, but this

has not been generally accepted, partly because it

involves a reduction by a factor of 12 in the number of

observations from which each maximum is obtained

and, hence, concern that the asymptotic distribution

might not be appropriate. Estimates of either or both

the seasonal or directional variation of extreme values

(e.g. what is the N-year Hm0 for northerly waves) may

be possible, but suffer from decreased confidence due

to shorter data sets. This may result in higher return

values in certain directions or months than the all-data

analysis.

The choice of distribution to fit the upper tail is in

general limited to those given in Table 9.1. Theory gives

asymptotic distributions for the upper tails of any

distribution, analogous to the extreme value theory. For

example, distributions whose maxima are distributed

FT-I have an upper tail asymptotic to a negative

exponential distribution (Pickands, 1975). However, in

practice, there is the problem of determining where the

upper tail commences — which determines the

proximity to the asymptote — and the theory has not

proved useful, to date, for analysing wave data.

Sometimes, instead of analysing all values above

some threshold, a “peaks-over-threshold” analysis is

carried out, analysing only the peak values between

successive crossings of the threshold. Usually the

upcrossings of the threshold are assumed to be from a

Poisson process and the peak values either from a negative

exponential distribution or from a generalized Pareto

distribution (see NERC, 1975; Smith, 1984).

Sincerely,

Oleksandr.

Thanks for the answers. 

What I concluded now is that, there is not a limiting and totally accepted proof for almost non of the distributions. It is rather matter of past studies and getting same results of distribution in a specific field or event.

Therefore, used distribution function of any kind mostly depends on the behavior of the sample and the goodness of fit through tests like Chi-square and Kol.-smir. etc. It is still favorable to categorize all of them in a chart with different aspects and applications in the literature. I think it would be a title of complete handbook.

Note:

• Dear Oleksandr. Thanks for the PDFs that you attached they are very precious ones. The referred literature are helpful too, I will definitely read them all.
• Dear Kirill. It was nice link to the Wikipedia, but I rather care about the physical evidence and explanations than mathematical properties of them. You know what, in hydrology (My field) first you have to decide on the nature of the phenomena and then apply for finding a solution. It is still hard to define the true properties of natural behaviors. 
• Dear Andrew. I think you concluded it all, but there is still some circumstances about choosing the right one. :)

Thankfully yours

There seems to be a lot of confusion abut what actually constitutes a probability distribution. Before we can even attempt to answer the question of what "physical meaning" is held by a distribution we need to actually define what a distribution is.

A probability distribution is simply a method of assigning probabilities to the sets of possible values of a random variable. If I tell you that the probability that the random variable X takes on a value between, say, 3 and 5 is 0.8, and then the probability that the same random variable takes on a value over every other set of the real numbers is 0.2, then I have told you the probability distribution of that random variable. 

More generally, say you have two random variables: X and Y. If it is true of EVERY set of real numbers that the probability that X takes on a value within that set is the same as the probability that Y takes on a value within that set, then X and Y have the same probability distribution. 

Now, as you can guess from this explanation, the term "probability distribution" is very general and aspecific. When we talk about, say, Normal distributions, or binomial distributions, or any of the other "named" distributions you use in statistics, we are actually talking about "probability density functions" (in the case of continuous random variables) and "probability mass functions" (in the case of discrete random variables). These are ways of characterizing specific special cases of probability distribution. Formally, a probability mass/density function is the Radon-Nikodym derivative of the measure implied by a random variable with respect to the Lebesgue measure (in the continuous case) or the counting measure (in the discrete case).

Now, this is a heavily theoretical definition, I admit, and relatively inaccessible, but I bring it up to emphasize that these are RELATIVE measures, NOT absolute measures. That is, a discrete probability is defined as the probability of a random variable being equal to a specific value (i.e. the probability is determined relative to the "counting measure"; e.g. P(X=1)=0.5), where a continuous probability is defined as the probability of a random variable taken on a value within an infinitely small range (i.e. the probability P(X=1) in the continuous case is technically undefined, we approximate it with the limit of an arbitrarily small interval). 

As for your comment that, "there is not a limiting and totally accepted proof for almost non of the distributions." Strictly speaking, this is false. Most distributions can in fact be proven from first principles. The simplest cases would be the Bernoulli and Binomial distributions, whose probability mass functions can be proven by exact enumeration. For example, when you flip a coin, what is the probability of heads? Of course it is 1/2 (assuming a fair coin). What is the probability of two heads if you flip the coin again? Likely, you can correctly answer 1/4 without even having to think about abstract probability functions. The answer is 1/4 because, with two coin flips, there are four possible outcomes: HH, HT, TH, and TT, of which only one meets the two heads criteria. So here we have assigned a probability (1/4) to one subset (HH) of all possible outcomes (the set of HH, HT, TH, and TT). Well, what happens if we flip the coin 10 times and are interested in the probability of there being 5 heads in that sequence? Well, we know that the probability of getting 5 heads is (1/2)*(1/2)*(1/2)*(1/2)*(1/2)=(1/2)^5=1/32. However, 1/32 isn't the correct answer; we also need to factor in the probability of the other 5 coins NOT being heads (i.e. being tails), which is also 1/32, and we further need to factor in the number of different possible ways for a sequence of 5 heads and 5 tails to be produced (e.g. HHHHHTTTTT, HTHHHHTTTT, etc.). The answer then becomes (10 choose 5)*(1/2)^5*(1/2)^5, which is the probability mass function of the binomial distribution. If you sit down and write down all of the possible 10 coin outcomes and calculate the number with 5 heads, you get the same answer as using this function. 

To show that this is a RELATIVE probability, note that the set of possible outcomes includes HT and TH for the 2 flip case, which we are classifying as separate outcomes. That is, we are implicitly assuming that the ORDER of outcomes matters. If we aren't interested in order but only the raw totals, HT and TH are the same outcome. Since we no longer care about order, our method of calculating the total number of possible outcomes changes, since we are collapsing the HT and THs into the same outcome (and so on for the 10 flip case). That is, the probability is relative to the measure we are using to determine and weight the possible outcomes. 

Similar proofs can be completed for most probability mass/density functions, with varying degrees of complexity. See the attached link to a .PDF file proving the density function of the Normal distribution from first principles for an example. 

Hopefully, this explanation has given you a somewhat clearer idea of what a probability distribution is (and, more specifically, what a probability density/mass function is, which is really what the thrust of your question is directed towards, rather than the more general probability distribution), let us try and answer your question regarding "physical meaning" or "physical definition."

In reality, there is no strict physical definition of any probability distribution or probability density/mass function, since these terms are actually describing PROCESSES rather than discrete phenomena. A distribution is the PROCESS by which we, according to some set of rules, assign probabilities to outcomes. These probabilities are always relative to our measure of determination; as already discussed, the probability we assign to outcomes of a series of coin flips depends on whether or not we believe the order of flips to be important (i.e. whether heads then tails, HT, is a concretely different outcome from tails then heads, TH). When we define distributions in this way, the only physical interpretation is in the system you are using to define the outcomes. That is, a binomial distribution can be used to define coin flip probabilities as we have shown, but it can also be used to describe voting results, or die rolls, or any real world situation where you can define discrete outcomes (win/loss, yes/no, etc.). These are vastly different "physical" interpretations, yet all rely on the same basic probability mass function and attending mathematical theory. 

Any physical interpretation is strictly the result of the context of the experiment and measurement you are modeling using these probability distributions. To stick with the example of the binomial distribution, you are implicitly assuming that whatever process you are modeling has a discrete binary outcome (e.g. yes/no, win/loss, heads/tails, etc.), that each "trial" (e.g. coin flip) is independent and does not impact the probability of success in any other trial, the success probability is constant across all trials, and that the order of successes does not matter (e.g. HT and TH are the same outcome). 

For another example, let's go with the Normal distribution. Here, you are assuming that symmetry; i.e. the probability that the unit you are measuring takes on a value greater than its mean is equal to the probability that it takes on a value less than its mean. You are also assuming it is unimodal; i.e. that there is a true mean, and that the distribution can be fully specified knowing only the mean and the variance of the sample. Note that these assumptions are not unique to the Normal; in fact, there are dozens of different distributions that under the right conditions will produce bell curves that simulate the same behavior. The reason we choose the Normal distribution is because of its mathematical properties (and, in fact, it can be shown that the Normal has maximum entropy of all probability distributions over all real numbers with a specified mean and variance). 

So, in short, in and of themselves probability distributions have no physical interpretation. They are mathematical constructs defining a process by which we assign probabilities to certain events within a broader space of possible events. Any physical interpretation is inherent to your choice of events and process, not the distributions themselves. The question of how to choose the "correct" distribution is complex, with no easy answers; it depends on the balance you choose to strike between fully accurate modeling (i..e overfitting) and mathematical convenience. For example, instead of using any standard probability density function, you could always directly determine a unique function for your data that would perfectly describe the behavior of that data. However, this would overfit the data, and you wouldn't be able to make any inference out of it, only crudely describe its probabilistic behavior. This is the equivalent of writing out every possible outcome from the 10 coin flips example mentioned previously and then just counting the number of times your outcome occurred; it does perfectly describe the data that comes from flipping a coin 10 times, but has no generalizable properties that are inherent to classifying the process instead using a closed form probability mass function.

http://www.ncssm.edu/courses/math/Talks/PDFS/normal.pdf

"See the attached link to a .PDF file proving the density function of the Normal distribution from first principles for an example."

Thanks for this link!

I even found a similar YouTube-video with this derivation:

https://www.youtube.com/watch?v=ebewBjZmZTw

Intersetingly, the normal distribution has many and quite different derivations. See here for further references:

http://www-biba.inrialpes.fr/Jaynes/cc07s.pdf

That's a great reference, Jochen. In fact, it is true that the different probability distributions can largely be derived in a number of different ways under different theoretical frameworks. This is, of course, where statistics and mathematics starts to get very complicated and confusing very quickly. Just for probability distributions, we have probability theory, measure-theoretic theory, set theory, real analysis, ... and on and on.

Ryan

Your explanation was very informative and good. 

Let me explain what was the main problem (if you are interested!!). The idea arose when I was investigating the characteristics of rainfall over different places of the world. As a short memory and low persistent procedure we need to do some calculations and modeling (e.g. Return period, risk analysis, Box-Jenkins etc.) about the rainfall. Looking back to literature, different research makers used some distributions like Log Pearson type III, Gamma, Logistic etc. When I tried some of them on some sample data set, results were totally different (even for regions that geographically similar). Then I began to ask questions about what characteristics of a certain distribution is helpful in primary and fundamental decision making and  finding a general solution or at least categorizing some of them due to some features and properties.

Note: If you know any reference that explains such nature and informative characteristics of all distribution please let me know.

Again I want to thank for your time and attention.

Dear Babak,

Your last message has given a reason for me to make some remarks about the rainfall distribution mentioned by you.

There was a good discussion about rainfall in the RG question "What are the possible relationships between the number of wet days and the rainfall concentration?" asked by our colleague Robert Monjo. The link is provided below.

https://www.researchgate.net/post/What_are_the_possible_relationships_between_the_number_of_wet_days_and_the_rainfall_concentration

I have also attached the most interesting papers related to (as PDF files).

I think that it can be helpful for you to read the articles of Javier Martin-Vide and Robert Monjo. The links are also provided.

https://www.researchgate.net/researcher/79600077_Javier_Martin-Vide

https://www.researchgate.net/profile/Robert_Monjo/publications

The researchers mentioned developed some informative characteristics for the rainfall distribution (the first one is developed by Martin-Vide and the second one by Monjo):

concentration index (CI) that evaluates the varying weight of daily

precipitation, i.e. the contribution of the days of greatest rainfall to the total amount;

index “n”, defined from the exponent obtained when adjusting IDF (Intensity-Duration-Frequency) climatic curves to MAI (Maximum Average Intensities) curves. That index provides information about how

maximum precipitation is achieved in a certain climatic area, according to the relative temporal distribution of maximum intensities.

Sincerely,

Oleksandr.

P.S. I enjoyed the articles of Javier Martin-Vide and Robert Monjo very much. I think you will find them very interesting for your research.

I think these papers contain "characteristics of a certain rainfall distribution which is helpful in primary and fundamental decision making and finding a general solution or at least categorizing some of them due to some features and properties" (Your note considered: "If you know any reference that explains such nature and informative characteristics of all distribution please let me know".).

I believe that you are on the right way to get a full and deep knowledge of your problem.

Babak,

Since this is not my field, I can only offer very general advice. From a cursory glance, it seems that the sources provided by Oleksandr will be very helpful to you.

Let's look at the three distributions you listed as examples and provide some general notes on them.

Log-Pearson Type III: this distribution is, in fact, a log-normal distribution with a non-zero skewness coefficient. As I mentioned previous, a normal distribution (by consequence of the central limit theorem) can be interpreted as the sum of a series of independent random variables. If we interpret this theorem in the log domain, we can say that a log-normal distribution is the PRODUCT of a series of independent random variables. That is, we might interpret any log-Pearson type III process as being the result of a series of independent factors acting on one another multiplicatively; the skewness coefficient then tells us that these forces are not acting symmetrically (e.g. some are acting with more "force" than others). In addition, since it is operating on the log scale, there is an implicit assumption that the generative process admits outliers; that is, there will be instances of extreme (with respect to the central tendency of the distribution) rainfall. Another potential assumption related to a log-normal process is Gibralt's law: that is, the growth rate of the process is independent of the size of the process. However, you should investigate the literature in our field on the applicability of this assumption, as it is one that is mathematically disputed.

Logistic: A lot can be said on the logistic distribution, but in a practical sense it is primarily used because, under certain regularity conditions, it can be used to approximate a normal distribution. A log-logistic distribution will, under certain parameterizations, be equivalent to a log-normal distribution (which, as we just saw, is a special case of the log-Pearson III). As mentioned previously, this implies the presence of outliers and asymmetric growth rate (i.e., if the skewness is positive, you are saying that the growth rate rises initially, and then decreases; vice versa if the skewness is negative).

Gamma: As with the logistic distribution, the gamma distribution is utilized primarily for its utility rather than anything inherent to its interpretation. The gamma is useful because it can assume a variety of different shapes/functional forms, but still only requires the specification of two parameters. It is extremely flexible. Under certain conditions, a gamma can be used to approximate a Normal distribution, but with the restriction that it is only defined over POSITIVE numbers. And, in fact, a gamma distribution is a type of Pearson III.

So, these three parametric models are, in effect, making the same set of assumptions about the data. There are some subtle differences between the three, but in practice they are simply different estimation strategies for the same basic process.

Oleksander, thanks for the links and PDFs they are amazing.

I am investigating on annual rainfall and that is why They should show some similar patterns. As you know, in annual patterns there should be a general procedure free from seasonality and daily changes. 

Daily rainfall is much complicated and have much stochastic nature. I will read them all as soon as possible.

Ryan, this is a good start. That is exactly what I am looking about. This kind of discription and analysis is a way to recognize the pattern. I hope to find a general manipulation thechnique soon. Thank you.

It has been shown that the log-normal distribution, which is often applied as a simplified model for particle sizes, can be derived (*) trough a kinetic based reasoning, considering time spent for particle growth or shrinking as normally distributed. Moreover, particle growth rate (negative for shrinking) is taken as linearly dependent on particle size, while the particle size is (unrealistically) unrestricted by either a minimum or a maximum size. Even if these assumptions are crude, the model fits reasonably well monomodal distributed particle assemblies at many practical situations. However, neither nucleation, coalescence, breakup, interfacial chemical reactions, Ostwald ripening, nor the mixture of distinct particle assemblies, are specifically accounted for by this highly simplified model. Therefore, deviations may be quite important, and more complex models have been proposed. Some are empirically adopted, without a sounding mathematical-physical justification. This was also the case for the log-normal distribution, which was adopted earlier for many years without known theoretical justification.

(*) Riyad R. Irani, Clayton F. Callis, “Particle Size: Measurement, Interpretation, and Application”, John Wiley & Sons, New York and London, 1963.