A parameter is defined as a value of the population whereas a statistic is a value of the data, i.e., the mean can be a statistic or parameter. However, it can be quite ambiguously and the context is obscured or missing . For example, the linear model has parameters B0 and B1; in E(y|x) = B0+B1*X1. If B1 is estimated via ML, B1 is a statistic (likelihood), but via MCMC a parameter (posterior). However, the coefficients B0 and B1 are described as parameters, even before estimation, but are not population parameters.
Thus, in referring to the parameters of the linear model the ambiguity remains in the use of the word parameters (if one does not specify the method of estimation or is unaware of it). Given I would like to express in a single sentence what I did, "I estimated the model statistic B1" highlights that it is the likelihood of the coefficient B1, whereas "I estimated the model parameter B1" highlights that it is the posterior of coefficient B1.
The question, is it valid to refer the the coefficients B0 and B1 after estimation as statistic or parameter given the likelihood or posterior? I have not read the expression statistic(s) in referring to the coefficients in a linear model only as parameter(s) (or sample parameter). I am trying to avoid ambiguity in my writing and clear my brain.
Thank you in advance
P.S. I adjusted the question
Ette Etuk Yes, but the word parameter here leads to ambiguity with the population parameter. The estimate resulting from the ML is not the population parameter but a sample parameter, thus a statistic. In my opinion it would be be less ambiguous to suggest we estimate the coefficient of the model resulting in the statistic (likelihood). Or we estimate the coefficient of the model resulting in the parameter (posterior). If I only mention I estimate the parameter, it would result in the understanding that I estimated the posterior. You last sentence makes sense and this is what I would refer to. But, it is used intertwined. Why use the word parameter in the linear model for the coefficients, but also for the estimate of the population this would lead to confusion when not explicitly mention if we do not talk about the coefficients or about the estimation of the coefficients. My question to you in the following sentence about what am I talking: "the parameter B1 in E(y|x)=B0+B1X1 is 1" Do I now refer the the model parameter B1 or to the coefficient as estimation on the population?
Yes, I understand this, I think my question was unclear. The question is not about MLE. It is about describing that the the coefficient B1 was estimated as a sample statistic or as a posterior parameter, by referring to B1 as being a statistic or parameter, that's what I mean.
I can understand the sample statistic contains information on the population it is however not that ML estimates population parameters. This is not what you mean I presume? If this is not what you mean, how can MLE result in a population parameter or do you mean to say that MLE results in the estimation of the sample statistic?
To my understanding you maximize the data given the model. The MLE is about the data P(Data|parameter) or L(parameter|Data) and solely in the long-run addresses the population parameter (if you believe so), not as individual estimate. The posterior suggest something about the parameter given the data P(parameter|Data). By describing the estimate obtained by ML it seems implied the that the ML estimate is the posterior parameter and not the likelihood of the parameter.
You are completely correct and I appreciate your patience. But I think we talk past each other. Based on the responses it seem that I do not formulate my question properly and perhaps no-one else understands it. The question is, not what a parameter, statistic, population or sample is. The question is whether we can reduce ambiguity to referring to coefficient B1 as a statistic, parameter or oompa loompa?
If the statistic refers to be obtained from the data, and the parameter to what this might be in the population. The response "The use of ML estimate of the mean refers to the mean as a parameter." refers to the mean as an estimate of the population parameter. However, to my understanding the MLE does not result in the estimate of a population parameter as "single event", it is a statistic obtained by maximizing the data to fit it to a model. We cannot treat the MLE as a "single event" estimate. This response shows the ambiguity of solely using the word parameter. The response "A sample statistic is an estimate or estimator of a population parameter" highlights the same misunderstanding. The likelihood is not directly about the population parameter as is the posterior and remember this to be referred to as the "likelihood fallacy" in:Article The Fallacy of Placing Confidence in Confidence Intervals
.Indeed a sample statistic is an estimate of the population parameter. But whether we treat the parameter as fixed or not and obtain a likelihood estimate or posterior estimate, makes a big logical difference. Therefor my question was if there is a less ambiguous wording to address if the estimate of B1 as statistic (which I refer to as a likelihood estimate), parameter (which I refer to as a posterior estimate) or oompa loompa (which I refer to as the estimated meaningless of this response and life itself). Not whether a statistic is resulting from a process of estimation . By using this single word to the estimate of coefficient B1 it would become immediately clear that B1 is a likelihood estimate or posterior estimate, without sifting through text to find out how one estimated B1.
Wim Kaijser In your example of the linear regression model, the coefficients (or called parameters) B0 and B1 are usually estimated by least squares principle, which is probably the same as MLE. I think this is based on the frequentist view where the parameters are treated as "fixed" constants but unknown. On the other hand, I think estimating B1 via MCMC is based on the Bayesian view where the parameter is treated as a "random variable". As a practitioner in measurement science, I have long been puzzled by the Bayesian view and treatment. I finally figured it out, the two different views and treatments are due to different frameworks of reference. In measurement science, I think the two views can be exchanged or transformed. I proposed a frequentist-Bayesian transformation (please see: Preprint A new modified Bayesian method for measurement uncertainty a...
). So, my point is that these two paradigms (frequentist and Bayesian) are confusing in practice. I hope my response doesn't add more confusion.
It is worth reading what Fisher himself said when introducing the term parameter " He introduced it, along with many other terms, in "On the Mathematical Foundations of Theoretical Statistics", Philosophical Transactions of the Royal Society of London, Ser. A. 222, (1922) 309-368. Parameter arrived with statistic for Fisher saw the need for two terms (p. 311):
it has happened that in statistics a purely verbal confusion has hindered the distinct formulation of statistical problems; for it is customary to apply the same name, mean, standard deviation, correlation coefficient, etc. both to the true value which we should like to know but can only estimate, and to the particular value at which we arrive by our method of estimation ...
With the new terms Fisher could write, "Problems of Estimation ... involve the choice of methods of calculating from a sample ... statistics, which are designed to estimate the values of the parameters of the hypothetical population." (p. 313) He would recall, "I was quite deliberate in choosing unlike words for these ideas which it was important to distinguish as clearly as possible." (letter (p. 81) in J. H. Bennett Statistical Inference and Analysis: Selected Correspondence of R. A. Fisher (1990)). Parameter did not replace any existing standard term. Fisher had used "arbitrary element" (1912) and "population-character" (1921), while Karl Pearson's "frequency-constant" could mean either a parameter or a statistic. While Fisher's use of statistic was criticised (See entry on Statistic), parameter fell in with an established usage, which the OED2 traces to the mid-nineteenth century, viz. "a quantity which is constant (as distinct from the ordinary variables) in a particular case considered, but which varies in different cases."
Source: https://mathshistory.st-andrews.ac.uk/Miller/mathword/p/
"In statistical theory R. A. Fisher used statistic to refer to a quantity derived from the observations--before settling on it he had used "statistical derivative" (1915), "derivate" (1920) and "statistical derivate" (1921). Fisher presented the new term in his "On the Mathematical Foundations of Theoretical Statistics", Philosophical Transactions of the Royal Society of London, Ser. A., 222, (1922), 309-368: "These involve the choice of methods of calculating from a sample statistical derivates, or as we shall call them statistics, which are designed to estimate the values of the parameters of the hypothetical population." (p. 318) The term parameter was also new and with statistic the two made a pair. (See the entry on parameter for Fisher's reasoning.) Fisher called the statistics arising in estimation problems estimates. He had no name for statistics arising in testing but since the 1950s they have been called "test statistics." Fisher's term was not well-received initially. Arne Fisher (no relation) asked him, "Where ... did you get that atrocity, a statistic?" (letter (p. 312) in J. H. Bennett Statistical Inference and Analysis: Selected Correspondence of R. A. Fisher (1990).) Karl Pearson objected, "Are we also to introduce the words a mathematic, a physic, an electric etc., for parameters or constants of other branches of science?" (p. 49n of Biometrika, 28, 34-59 1936)."
Source: https://mathshistory.st-andrews.ac.uk/Miller/mathword/s/
Hening Huang Thank you. Yes, you are quite right MLE results is the same value as least squares. I found the most simplest, shortest and easiest of examples in "Linear regression: ... " by Stone, James V.
I personally tend more to the Bayesian view although I sympathize with the frequentist view. Hence, I do not want to update my personal believes. But, expression of probability and uncertainty is not solely a thing of mathematics, the data and the long run. The issue in ecology is that every sample is a biased (in too many forms) and each year is different. The parameter of the dodo (the extinct bird was once) >0 and now 0. So the population parameter is not fixed. Then again, when performing an laboratory experiment assessing the LD50 of substance X on Daphnia I feel quite "confident" that repetition in the long run might serve very well for a "good" experiment.
Not confusing at all, I look forward to reading it.
At the risk of more confusion than light. I conceive of a (population) as having an unknown value (and this is the contentious bit) which is unknowable - a theoretical entity outside of a simulation study where you have played God.
Consequently, for me, the mean or median of a posterior distribution in a Bayesian analysis is also an estimate.
Kelvyn Jones Many thanks for your elaborate response! One interesting thing that pops out to me in "Problems of Estimation ... involve the choice of methods of calculating from a sample ... statistics, which are designed to estimate the values of the parameters of the hypothetical population." is the wording hypothetical population. Ian Hacking also addresses in "Logic of Statistical Inference" that Fisher saw the long run more hypothetical (I cannot recall the page number!). Consequently and completely speculative, I draw a connection to this article as Fisher becoming more open to "subjectivity" Article R. A. Fisher on Bayes and Bayes' theorem
. Pearson saw it more as "real" i.e., randomly drawing 30 bottles of beer from a production conveyer to estimate the volume. In any case, this somehow makes me think of an issue in context and language where words or often differently use in practice than theory (https://en.wikipedia.org/wiki/Ordinary_language_philosophy). This makes me think that if estimate is for both likelihood and posterior statistic accounts also for both schools of thought. Yet, the context might stay obscure. Just some random noise popped up.At your latter response, I see no problem there. The part "which is unknowable" is popping out for me. Is the unknowable fixed or or is it unknown? where I would say unknown. Knowing it is fixed would already assume some knowledge about the behavior. I am curious, which argument would be used to treat it as being fixed or not fixed?
Best,
It is confusing only to you to be honest
Also it feels like you are making such a fuss over something very trivial
Words have the meanings you asign to them
When Fisher "invented" the word "statistic" it became object of controversy - and I personally don't like it and try to avoid it - but the difference is very clear
The idea of a parameter being fixed or not is trivial to be honest
You say you might know something abut the behavior of a parameter once you know it is fixed
You are actually mixing up Neyman/Person theory and Bayesian
If you have a reference group (you may call it population if you like) such as the number of people in the world that have diabetes, I personally do not see why it'd would be hard to believe it is a number that is both fixed and unknown
Sometimes things in Statistics sound counter intuitive or made up but that's because it IS made up
Neyman-Person statistics considers parameters fixed while bayesian statistics sees them as random variables but that's just a starting point as to how to assess variability
Nothing forbids that a few years from now somebody might come up with a new apporch which could be a mixture or even something completely unrelated
Stefano Nembrini : it was not Fisher who invented the word statistics. Credit must go to Sir John Sinclair (https://en.wikipedia.org/wiki/Sir_John_Sinclair,_1st_Baronet) 1754-1835, who introduced the term into English. He adapted it from a German word. To quote the man himself
"Many people were at first surprised at my using the words "statistical" and "statistics", as it was supposed that some term in our own language might have expressed the same meaning
…
the idea I annex to the term is an inquiry into the state of a country, for the purpose of ascertaining the quantum of happiness enjoyed by its inhabitants, and the means of its future improvement; but as I thought that a new word might attract more public attention, I resolved on adopting it, and I hope it is now completely naturalised and incorporated with our language."
And I would urge you to be careful with your own words. It does not "feel like" the questioner is making a fuss. This is something that you think, an opinion. Thoughts are not feelings.
Nor do words have the meanings that you assign to them. If they did, communication, science, ethics, even food labelling would be impossible.
Finally, if the question irritates you, I suggest you leave it alone. Researchgate is a collaborative, supportive platform. If you want something more adversarial, there's plenty of choice.
We estimate parameters used in a model for the data. In Bayesian methods the parameter estimates represent a state of knowledge. In frequency of occurrence methods the estimates represent a state of nature.
If the model for the data does not map onto the data in a one-to-one mathematical relationship, the parameter estimates are flawed.
The principles and procedures used to compute the parameter estimates and their uncertainties are the statistic.
I said statistic not statistics
Which is just the what not the how (quantity vs procedures)
Parameters are variables used to define a family of functions. The function y(x) = 3x+2 is a single straight line with intercept 2 and slope 3. One can generalize the "line" function by using parameters for the intercept and the slope. So one can talk about a family of lines that differ in slopes and/or intercepts. Similarily, the function y(x) = 2*sin(x-3) describes a sine wave with amplidue 2, frequency 1, and offeset 3, which can be generalized to a family of sine waves when amplitude, frequency and/or offset are described by variables which are then called parametes.
One can use the same defintion to generalize a distribution of a random variable. The function 4^x*exp(-4) / x! for instance is the probability mass function of the Poisson distribution with the expectation 4. Again, we may want to talk about Poisson distributions in general, that is, the family of Poisson distributions for which we give the expectation as a parameter (typically denoted with lambda, λ).
We are certainly free to re-parametrize the Poisson distribution in order to express the expectation by a familiy of a different function. Lets say we want to express that the expectation linearily depends on the value of some variable v, but we do not want to give a particular intercept and slope. So we simply substitute λ with the (parametric) function λ(v) = β0 + β1*v. Instead of λ we now consider β0 and β1 as the parameters of the Poisson distribution.
In statistics we use data to estimate values for β0 and β1. The estimates are determined in a way to make the data least surprising (or maximal likely). Concrete estimates of concrete data are: numbers. It might be -3.4 and 0.52. But as usualy, we often want to talk about this more generally. So for some given set of data we get get some estimated values, and we distinguish these estimated values from the parameters by either placing a hat above the Greek letter or by using an corresponding latin letter: like b0 as the estimated value of β0 and b1 as the estimated value of β1. Note that b0 and b1 are not parameters; these are estimated values.
To make thinks slightly more complicated, the entire function to obtain the estimated values are called estimators. Thes functions have particular statistical properties, like efficiency, unbiasedness etc. I don't know if there is any particular naming convention for estimators. I have seen authors using the same symbols as for estimated values. This may be a source of confusion about estimated values and estimators (not sure if this is relevant), but not between parameters and estimated values.
A parameter, say "mu", is a "real" quantity (on the real line) that is UNspecified in a model (given by a formula); it becomes specified when WE assign to it a "real number".
When we measure a characteristic of the items of a physical sample we get some "real numbers" which are the "determinations" of a Random Variable, say X, representing all the possible measures we can collect.
If the sample size is n, then we collect n numbers, x1, x2,...., xn, each one being a determination of the Random Variables X1, X2,...., Xn, identically distributed as X.
The "real number" m which we compute from a suitable formula, containing the determinations x1, x2,...., xn, is named the "estimate" (or "statistic") of the parameter "mu".
The "estimate" m is the determination of a Random Variable M which mix, with a suitable formula, the Random Variables X1, X2,...., Xn; M is named the "Estimator" of the parameter "mu".
The "Estimator" M has some statistical properties such as correctness, unbiasedness, efficiency, minimum variance, ...
We assign the same properties to the "estimate" m.
There is NO confusion, at all.
The confusion arises with "Bayesians" who name parameter what actually for them is a Random Variable, to which they associate a probability distribution!!!!
William C Hutton this sounds easier, I might start using this explanation. It is far easier to communicate than talking about, likelihoods, posteriors, priors, long-runs, single cases and updating prior believes, thanks!
Jochen Wilhelm Yes, I noticed that many authors exchange them, this is rather confusing. I tend to use β but perhaps I should switch to b? The question is would this be valid or would I step on someones toes by doing so?
Wim Kaijser , whatever you do, you will always step on someone's toes :)
Anything is ok as long as you clearly state what symbols you use for what purpose, so that any reader can understand and follow without being confused.
However :) many readers will be used to some kind of convention. You take an additional mental burden on the reader by breaking with these conventions. I would avoid this unless there is a good reason to do so.
Just my 2ct
PS: These goddamn Bayesians... ;)
Wim: I‘m glad you found this explanation useful. In my opinion, linking parameters to the the model for the data does not get emphasis. A parameter and its estimate are different. Likewise, the implications of an incorrect or incomplete model does not receive the proper attention.
A parameter is a number describing a whole population (e.g., population mean), while a statistic is a number describing a sample (e.g., sample mean). The goal of quantitative research is to understand characteristics of populations by finding parameters. In practice, it’s often too difficult, time-consuming or unfeasible to collect data from every member of a population. Instead, data is collected from samples. With inferential statistics, we can use sample statistics to make educated guesses about population parameters.
Your example is a linear regression model (or equation) and B0 and B1 are (called) parameters in the field of hydraulics (as far as I know). We use data to estimate these parameters and we do not use the term "statistic" at all. Indeed, as you stated, "the coefficients B0 and B1 are described as parameters, even before estimation, but are not population parameters."
- Badges
- Science method