Evaluation of MLR model quality is often based on comparation between the values of R-square and R-square of prediction, to find whether the model is acceptable, whether fit data well. I suppose that acceptable model should have R-square of prediction greater than 0.5, or more grater for prediction. However In practise I often meet insufficient R-square of prediction. What is your opinion on the models, that R-square of prediction is less than 0.5? But other characteristics as R-square, residual normality, ANOVA p-values, homogeneity of variance are okay. Are these models in particular useful for the interpretation of the experiment? Suppose that it is a CCD-faced (i.e. full factorial design) with 3 levels with 3 variables, and triplicated center measurement.
Hi, Petr,
First, by predictive R2 I understand that the program deletes one observation at a time and tries to predict it by fitting the model to the other data. I could imagine several possible explanations to your observation. Here are two of them. A pessimistic one is that your model "artificially" fits the data because it has so many parameters, and not because the model reflects the underlying relationship. This could of course lead to bad predictions. But the predictive power of your model need not be bad in order to give you a predictive R2 substantially lower than the ordinary R2. Instead it could be that when you delete one observation (point) from a CCD design (which is not a full factorial, but only contains 15 points for 11 mean value parameters, doesn't it, Petr?), you will lose very much information about some parameter, or parameter combination, for which this point was crucial. This could cause bad prediction of the deleted observation (except in the center point). In this case I would say your problem is due to the design in combination with the leave-one-out procedure, and that the predictive R2 is misleading. The real ability of the full model (fitted to all calibration data) to predict a new (additional) observation could be much higher than your predictive R2 indicates. I would then suggest you try a kind of separate test set: Make additional observations in some more points within or in the vicinity of the cube, which should not be among your design points, and see how well you can predict them.
Best wishes,
Rolf Sundberg
Hi Rolf,
Yes, prediction is performed by leave one out process from 15 (or 17 if triplicated) center points. I don´t know exactly if the center measurements are take to consideration, or averaged. I also guess, that number of variables for this type of validation is not too suitable. So it is motivation for me to design some external validation experiments. In order to maximally reducing of experiment I suppose three can be choice, what do you thing?
What is the response you are trying to model? It might need to be transformed.
I have seen R^2 (adj) and R^2(pred) come closer to each other when I remove certain terms from a model. You might have too many terms in your model and be over fitting it, as Rolf suggested. So, I would look at removing insignificant terms.
I would also look at the Lack of Fit for your model.
One other issue that I had was, the center points were too similar to each other. If the variability of these center points are too small you can have issues with prediction. How "bad" is you fit for each treatment combination in your design? Suppose that your confidence interval for each point is +/-0.001 and your fits fall within +/-0.003. If your acceptable levels are +/- 1.00, you don't have an issue. But, your R^2 (pred) will be off.
- Badges
- Science method
- Similar topics
- Mathematics
- Statistics
- Statistical Modeling
- Analyses of Variance
- ANOVA