Hello there, I got a response surface regression. In its ANOVA, the model p-value shows as 0.005 but the p-value of lack-of-fit shows as 0.038. Just curious how to determine whether the model is well-fitted. Or actually, those values have different meanings?
Hello Jianan Lin Very good question.
To determine whether the model accurately fits the data, compare the p-value (P-value) to your significant level. Usually, a significance level (also called alpha or α) of 0.05 works well. An α of 0.05 means that your chance of concluding that the model does not fit the data when it really does is only 5%.
- P-value < α : The model does not fit the data
If the p-value is less than or equal to α, you conclude that the model does not accurately fit the data. To get a better model, you may need to add terms or transform your data.
- P-value > α : There is no evidence that the model does not fit the data
If the p-value is larger than α, you cannot conclude that the model does not fit the data well.
I would clarify further on Abolfazl Ghoodjani 's reply, like this.
To find out if your predictor variables improve prediction of the data, you have to ask "compared with what?". So every model starts off with a baseline model that makes the same prediction for each observation – in OLS regression, this constant prediction is the mean.
The p-value for the model assesses whether the model improved prediction more than you would expect by chance – where improvement is relative to the one-size-fits-all, constant only baseline. So a significant p-value indicates that the predictors improved prediction more than you would expect by chance.
However, it does not tell you whether the model can be improved on. A linear model may help to predict a curved relationship, but the prediction errors will form a pattern, so you could have a regression model that under-predicts at the extremes and over-predicts in the middle. Such patterns mean that the model errors will not be normally distributed over their range. The p-value for lack of fit tells you that your model is not capturing some feature of the data – you are trying to draw a banana with a ruler, for example.
So you may have significantly improved prediction together with significant lack of fit in the same model.
Graphing your data is an excellent way of diagnosing the problem, and should really happen before you fit a model.
Does that make sense?
An incredibly good example from Ronán Michael Conroy
"you are trying to draw a banana with a ruler, for example"
- Badges
- Science topic
- Similar topics
- Mathematics
- Statistics