'Since the softmax distribution is not in the exponential family, it does not have finite
sufficient statistics.'
I cant understand what the concept of above statement is. Do you have any information in this field.
"A statistic is sufficient for a family of probability distributions if the sample from which it is calculated gives no additional information than does the statistic, as to which of those probability distributions is that of the population from which the sample was taken". (Wikipedia)
The sufficient statistic T(X), a function of the sample X=(x_1, x_2, ..., x_n), where x_j are iid, is connected to an unknown parameter Theta (either scalar or vectorial, multidimensional) which occurs in the distribution. Is it T(X) sufficient statistic for Theta, we can confirm e.g. by the Fisher-Nyman factorization theorem - check the www for.
You can start from a definition:
http://en.wikipedia.org/wiki/Sufficient_statistic
In a few words 'sufficient' means nothing more than what the exact word says.
Your softmax function:
http://en.wikipedia.org/wiki/Softmax_function
http://computing.dcu.ie/~humphrys/Notes/AI/boltzmann.html
Your question probably comes from:
http://www.cs.ubc.ca/~murphyk/Bayes/usage.22may00.html
"Fitting a softmax function can be done using the iteratively reweighted least squares (IRLS) algorithm. We use the implementation from netlab. Note that since the softmax distribution is not in the exponential family, it does not have finite sufficient statistics, and hence we must store all the training data in uncompressed form. If this takes too much space, one should use online (stochastic) gradient descent (not implemented in BNT)."
So, maybe it is a technical question about fitting and not a mathematical one.
Probably you should follow the guides of the last link...
.
Pittman-Koopman-Darmois theorem !
have a look here :
http://en.wikipedia.org/wiki/Sufficient_statistic#Exponential_family
and for the notion of "sufficient statistics", you can read the whole entry !
.
.
about "sufficiency", i have also found this old (and marvellously written) paper very useful :
Persi Diaconis
Sufficiency as statistical symmetry
Proceedings of the AMS symposium, 1988
(attached below)
.
Based on a sample if the statistic T is sufficient for the parameter Theta, then T contains as much information regarding Theta as does the sample.
A simple explanation can be found in the answer of your next question.
A sufficient statistic, is a way of condensing information provided by the data set into a smaller amount of data in such a way that we have not lost any information about the unknown parameter. For details please see the attached file.
u can see many example of sufficient statistics in following:
"احمد پارسیان،مبانی آمار ریاضی، انتشارات دانشگاه اصفهان"
All of the answers above are on target for defining sufficiency. The important question is why it matters to you. What is your research goal, and how would your results change if the statistics that you have are not sufficient?
- Badges
- Science topic
- Similar topics
- Mathematics
- Statistics