Degrees of freedom of a variable and the dimension of the linear space concerned are the same. Dimensions of two linear sub-spaces with only the null element common are additive, and so the degrees of freedom concerned are additive.
When n vectors of the type (xi, yi) are given, and we fit an 1-dimensional equation of the type Y = a + bX, in the Analysis of Variance table, 1 degree of freedom is allotted for regression. As the total degrees of freedom are (n - 1), degrees of freedom available for error variations are (n - 2). Similarly, using n vectors of the type (xi, yi, zi) if we fit a 2-dimensional equation of the type Z = a + bX + cZ, 2 degrees of freedom are allotted to regression, and so the degrees of freedom available for error variations are (n - 3).
The degrees of freedom for errors in the first case is (n - 2) not because there are 2 parameters in the model but because degrees of freedom are additive, and therefore we get the error degrees of freedom (n - 2) by subtraction of the degree of freedom due to regression from the total degrees of freedom (n - 1). Similarly, in the second example, the degrees of freedom for errors are (n - 3) not because there are 3 parameters in the model but because the degrees of freedom found by subtraction is (n - 3).
The number of parameters does not have anything to do here. If the model is 1-dimensional, the degree of freedom for regression is 1. If it is 2-dimensional, the degrees of freedom for regression are 2.
However, when a non-linear model with k parameters is fitted, in the analysis of variance, the degrees of freedom allotted to errors are shown as (n - k). Can that kind of an analysis be valid?
Dear Ehsan, I understand your plea.
I think when you say that you are "poor in mathematics" it is because you have not perhaps learned to derive the mathematics but you memorized formulas. I do not believe one can learn mathematics by memorization.
Mathematics is a language with a grammar that one must feel. I have never memorized anything in mathematics and took the time to derive even the simplest things that most had on their fingers by memorization but then they received a class--say complex analysis--where visualization and full understanding with "being part of" the unreal is important, all my classmates failed and I went through with flying colors.
If you want to improve your skills in mathematics, go back to the beginning of algebra 1 and make sure that you are able to "derive" all that is in there with formulas.
I will tell you a little story here, not to brag but to inspire you:
I learned mathematics to algebra in high school (when I was in high school they stopped at algebra 1) and when I went to college and I had to take the mathematics entrance test, which I did about 20-25 years after my high school, I had the same exact knowledge at the test as I did when I left high school. I have not lifted a single math book into my hands in the 20-25 years.
After the test I visited the Dean of math and asked her if I could advance faster than the normal since I was always very good in math and loved the subject and I wished it to be my major.
She gave me several books and 3 months. She told me "learn as much as you can on your own and we'll give you a different test" (she was a wonderful Dean to let me do that!).
I self learned algebra 2 through calculus everything in 3 months! She gave me a new test that I passed 100%. She grabbed my hand and with classes all full already, she took me in person to a calculus class in session and advised the professor that I am a special student and admit me. There was one seat in the room. I graduated with a degree of mathematics (UCLA) and though I was admitted to follow into the Master's degree, I left to another school to pursue PhD in Statistics--which I disliked and found non-creative and left for a different degree in a different field. At the end I still ended up with a PhD in a heavily mathematical field albeit it is in Neuroeconomics,, which is basically 2 fields combined into one.
This story is to show you how there is beauty in math and a ton of future in math as well. It is easy if you have the right frame of mind. So work on your "frame of mind"! That is what you need to improve and not your math skills or statistics. :)
Hope this will help you.
Degrees if freedom rises when we want to find a parameter fr a random variable, and because all of the regression parameters are random variables and we want to estimate them from a sample data, we should reduce the degrees of freedom depending on the number of estimations.
Dear Ehsan,
In the case of the simple Student's t-test for testing for the mean, we estimate 2 parameters: the population mean and the population variance. But the degrees of freedom would be (n-1), and not (n - 2) although there are two parameters involved.
Respected Professor
Really, very nice question I like it very much.
Dear Hemanta,
In the example you mentioned, if i explain more precise, the degrees of freedom is the number of samples- number of restrictions. Here the restriction is:
Sigma (x-mean)=0
So we have n-1 degrees of freedom.
Besides, if you estimate Xbar, you calculate variance (not estimate).
Dear Ehsan
That is precisely what I meant.
Sum of (x - xbar) = 0 is a restriction, a linear one. In fact, the equation ( or restriction, in your language) for estimation of variance is not counted here. Why?
In fact, the first restriction defines a linear sub-space of (n - 1) dimensions. That is why the degrees of freedom available to the variables is (n - 1).
The other restriction, the second one, does not define a linear space. Please observe this point. Had that too been linear, the available degrees of freedom would have been (n - 2).
It is not 'my example'. This is how the Student's - t is used. You have said that the degrees of freedom is decided by the number of parameters involved. At least in this example, that is not so.
You can not follow two different ways to define degrees of freedom. In the case of Student's - t, you have said that it depends on the number of (linear) restrictions. On the other hand, in the case mentioned in my question, you are saying that the degrees of freedom depends on the number of parameters.
In other words, you have agreed that there are in use two different definitions of 'degrees of freedom'. That is precisely the point I have raised. Can an analysis in which degrees of freedom are decided by the number of parameters be valid?
Please stick to mathematical explanations only. One of the two definitions has to be wrong. Either you accept that the linear spatial explanation is correct, and therefore the way of defining degrees of freedom based on the number of parameters is wrong, or you start demanding that the the theory of linear spaces and therefore the theory of Euclidean spaces are meaningless.
However, the theory of linear spaces is based on strict mathematical principles. Please do not forget that. I would like to state further that Statistics follows Mathematics; it is not the other way around, it never was.
Dear Hemanta
As far as i know in t-test ( i mean independent test) the degrees of freedom is:
n1+n2-2
it refers to the point that we have a restriction as i mentioned for each group. But in paired-test Df is n-1 because we want to test if the average pair difference is equal to zero, so here we have one restriction.
In the case of parameters as i mentioned before, they are assumed to be random normal variables, so estimating them needs another one restriction of zero mean deviation.
Therefore, in my opinion the definition of both ways are the same. The more the number of parameters, the more the count of mentioned restriction.
Regards
@Ehsan
is there a connection between no of parameters to be estimated and no of restrictions?
Dear Ehsan,
First, what you have mentioned is Fisher's - t, and not Student's - t. Secondly, in the case of Fisher's - t, there are supposed to be 2 populations from which 2 samples of size n1 and n2 are drawn. Once again, for the 2 sample means, there will be 2 linear restrictions which define 2 sub-spaces, and therefore for (n1 + n2) observations, there will be (n1 + n2 - 2) degrees of freedom.
I think, I need to explain one more point. Say, you have a sample of observations on rainfall in a place (or humidity, or price of wheat, or whatever variable you like to assume). The observations will be positive numbers. Positive numbers do not form a Group with respect to addition. To define a Linear Space, we would first need to define a Group. For that, we need to have negative numbers too in the set. Observe that (x - xbar) may be positive as well as negative, and sum of (x - xbar) is zero, just as the sum of all real numbers is zero.
Hence, unless you define xbar, you can not define (x - xbar). Therefore, defining xbar is a mathematical necessity. In other words, before being a 'property' of Statistics, xbar is a 'property' of Mathematics. When you have a sample of n observations, the degrees of freedom are (n - 1) already, irrespective of whether you go for the Student's-t test or not!
What I mean is: when you have a sample of 10 observations, you are in a situation to deal with 9 degrees of freedom anyway. Where is the question of 'parameter' here?
Finally, you seem to have very wrong ideas about the normal probability law and mean deviation!
Dear Hemanta
We estimate the population parameter mu using a sample which has a statistic 'MEAN'. This is the point that if we use samples then we have degrees of freedom. But, do not confuse between regression parameter and population parameter, although they both are assumed to be random normal variables. I speak from the view point of statistics.
In one sample t-test we have only one parameter to estimate.
Dear Subrata
As far as i know, yes there is. No parameters=No restrictions
Was the original question not that if the system of equations is linear then we have a linear type of degree of freedom and if the system is non-linear why we still use linear degree of freedom?
That is the way I understood it. Is that not what the question is? Perhaps I misunderstood something?
Dear Angela, in non-linear regression the degrees of freedom is the same for linear regression. Because the basic assumption of parameter estimate is the same for both ( zero sum of (X-Xbar)).
Dear Angela,
Indeed, in another question, I raised an objection that even when a non-linear model is used, the analysis applies R^2, the multiple linear correlation coefficient. The degrees of freedom for errors in the analysis of variance table are decided based on the number of parameters. This I found unacceptable.
You have mentioned something like 'linear type of degrees of freedom'. 'Degrees of freedom' is the same as the dimension of the linear space concerned. Do you mean to say that there is something like 'non-linear type of degrees of freedom'? What would be its physical significance?
No Hemanta, I did not mean that only based on the responses above I did not understand your question, which seemed clear in the email until I hit the post board here with what to me seemed unrelated answers. Like Ehsan above notes that "in non-linear regression the degrees of freedom is the same for linear regression. Because the basic assumption of parameter estimate is the same for both ( zero sum of (X-Xbar))" and I understand that but I thought this was your very question: "Is it OK to assume that in non-linear regression we calculate the degrees of freedom the same way as in linear regression?" I thought this was your question. Is it not?
Dear Angela,
No, my question was: can that kind of an analysis is valid when the degrees of freedom for errors are decided by the number of parameters? Please go through the description in the question.
OK Hemanta, thank you.
So then the question at the end of your paragraph of the original question you asked: "However, when a non-linear model with k parameters is fitted, in the analysis of variance, the degrees of freedom allotted to errors are shown as (n - k). Can that kind of an analysis be valid?" refers to what? If everything is decided by the number of parameters as you (and all books) suggest, then why would the (n-k) not be appropriate to find the degrees of error in a nonlinear model?
What I was thinking your questions was that in a nonlinear model where nothing is additive, why would we calculate the degrees of freedom as "additive"... that to me would have made sense. The way I read it now, I am not sure what the question is.
Dear Angela,
No, I am not suggesting that the number of parameters should decide the error degrees of freedom. The books may have decided so.
If the number of parameters should have decided the degrees of freedom, then in Student's-t test, the variables should have (n - 2) degrees of freedom, and not (n - 1) degrees of freedom. Similarly, in the Fisher's - t test, the variables have (n - 2) degrees of freedom, which is not geared to the number of parameters.
Additivity of degrees of freedom comes from the concept of additivity of dimensions of linear sub-spaces. The concept of sub-spaces can not be applied to models that are not transferrable to the linear form.
Why then do we have to decide the error degrees of freedom based on the number of parameters for a non-linear model? Is there any mathematical proof that it can be done? That was my question.
My question is a serious one. Just because it is written in the books accordingly, that does not mean that it has to be true. Where is the mathematical proof? Is there any book that includes a mathematical proof of this? As I had mentioned earlier, Statistics follows Mathematics; it is not the other way around, it must not be.
Suppose you have a model like Y = a.exp(bX) + c.exp(dX). There are 4 parameters here. This model can not be linearized. Will the error degrees of freedom in such a case be (n - 4)? If so, why? What is the mathematical reason? In fact, can there be a mathematical reason behind it?
That is a very good question dear Hemanta and I wonder if we have some major statistical expert in this group who can show some proof. I don't recall anyone every questioning the validity of this but I agree; there is much to be answered here.
Dear Hemanta and Angela, yes there is. As I said before no matter what the model is, the restriction is the same because here again all four parameters are normal random variable. Here total df is n-1, model df is (no. of parameters(k)-1), and so error df is n-1-3. The model always have k-1 degrees of freedom, because once 3 of the parameters estimated (in your model) the last one is calculated and NOT ESTIMATED based on other parameters, and as we know calculation do not reduce the degrees of freedom.
Dear Ehsan,
I have said earlier that your ideas about the normal probability law are imperfect. Please do not bring the normal probability law into this discussion. The parameters in a mathematical model, fitted using the method of least squares for example, are just parameters; there is no question of any normal probability law associated with the parameters. Please do not step outside mathematical logic. I do not understand what more to say.
Further, who told you that one of the parameters is calculated, and not estimated?!
Dear Hemanta,
If we do not assume the parameters as normal random variables, then there is no purpose of calculating PRESS (Predicted Residual Sum of Squares) in statistics. I talk in most times from the statistical view.
For your second question "Further, who told you that one of the parameters is calculated, and not estimated?!" I make a simple example. Suppose someone tells us to choose 4 numbers which it's sum must be 6, ( in non-linear regression it called Convergence Criterion and i just make it simple as sum of 6) so we estimate the first three as, for instance, 1, 3, and 1.5(sum=5.5). It's obvious that the last one MUST be 0.5 and it calculated easily. I shall say again that it was just a simple logical example.
Dear Ehsan,
You said that one of the 'parameters' is calculated, and not estimated!
Your example is not on parameters. It is on 'numbers' as you have yourself mentioned.
You are not accepting my question as a valid one. Whether you accept the validity of my question is a different matter. That does not mean that you would continue arguing in this way! Just say that my question is baseless; that is all. I would gladly accept that instead of learning statistics from you!
Dear Hemanta, My apologies
I'm not in a place to learn you statistics, I'm just a student and i do not mean to disrespect you or your mathematical proficiency. I just discuss what i learned to see if my ideas are wrong. Because I've learned statistics by my self and besides, I'm very poor at mathematics. It's a great pleasure to discuss my statistical viewpoint with a mathematician.
Sincerely
Dear Ehsan,
I am sorry. I am really sorry to have hurt your feelings! In fact, right from your first answer, I knew that you are a learner! I should have taken more care while answering. I am sorry.
If you have been learning Statistics on your own, then try to look into the statistical matters mathematically. You have mentioned further that you are poor in mathematics. Stop thinking in that manner. Mathematics is the easiest of all branches of knowledge. In everything else, you have to remember matters. In mathematics, you do not have to do that.
Ehsan, mathematics, like mother tongue, is unforgettable. It is unforgettable, because you do not have to remember it! Do not even think that you are poor in mathematics.
Do not be a slave of mathematics; make it your slave! Good luck!
Dear Hemanta,
No problem at all, it happen some times between a student and his master. I very appreciate your advices.
Dear Ehsan, I understand your plea.
I think when you say that you are "poor in mathematics" it is because you have not perhaps learned to derive the mathematics but you memorized formulas. I do not believe one can learn mathematics by memorization.
Mathematics is a language with a grammar that one must feel. I have never memorized anything in mathematics and took the time to derive even the simplest things that most had on their fingers by memorization but then they received a class--say complex analysis--where visualization and full understanding with "being part of" the unreal is important, all my classmates failed and I went through with flying colors.
If you want to improve your skills in mathematics, go back to the beginning of algebra 1 and make sure that you are able to "derive" all that is in there with formulas.
I will tell you a little story here, not to brag but to inspire you:
I learned mathematics to algebra in high school (when I was in high school they stopped at algebra 1) and when I went to college and I had to take the mathematics entrance test, which I did about 20-25 years after my high school, I had the same exact knowledge at the test as I did when I left high school. I have not lifted a single math book into my hands in the 20-25 years.
After the test I visited the Dean of math and asked her if I could advance faster than the normal since I was always very good in math and loved the subject and I wished it to be my major.
She gave me several books and 3 months. She told me "learn as much as you can on your own and we'll give you a different test" (she was a wonderful Dean to let me do that!).
I self learned algebra 2 through calculus everything in 3 months! She gave me a new test that I passed 100%. She grabbed my hand and with classes all full already, she took me in person to a calculus class in session and advised the professor that I am a special student and admit me. There was one seat in the room. I graduated with a degree of mathematics (UCLA) and though I was admitted to follow into the Master's degree, I left to another school to pursue PhD in Statistics--which I disliked and found non-creative and left for a different degree in a different field. At the end I still ended up with a PhD in a heavily mathematical field albeit it is in Neuroeconomics,, which is basically 2 fields combined into one.
This story is to show you how there is beauty in math and a ton of future in math as well. It is easy if you have the right frame of mind. So work on your "frame of mind"! That is what you need to improve and not your math skills or statistics. :)
Hope this will help you.
Dear Angela Stanton
Very nice, really you tasks are very much important and interesting. I am fully agreed with your work.
Dear All, as i've read your thoughts, i think in one thing we are in common: Do not use your memorizing skill in mathematics and statistics. I chose statistics because I hate memorizing. I'm not good at mathematics because in school times our teachers didn't tell us what is the use and importance o mathematics. Now I want to learn statistical modeling and now I'm understanding what is the use of algebra, integral, and differential.
I as a teacher of mathematics teach limits, derivatives as a kind of limits, etc. for first year students of one college in Krasnoyarsk, Siberia, Russia. I agree with the Angela statements.
Dear Professor Aleksey Zakharenko
Really, very nice to heard your answer.
- Badges
- Science topic
- Similar topics
- Mathematics