Dear Nor,

R² is, by definition, equal to (sum of squares explained by the model)/(total sum of squares). So, if you add an effect in your model, even if it's noise, R² will increase. So, R² is not necessary a good indicator for the quality of your model (you can build a model with R²=100% with only noise).

If we write: ESS=Explained Sum of Squares, TSS=Total Sum of Squares and RSS=Residual Sum of Squares then R²=ESS/TSS.

ANOVA tells us: TSS=ESS+RSS so: R²=(TSS-RSS)/TSS and then R²=1-RSS/TSS.

If you uses cross validation (usually leave one out), you can calculate PRESS (Prediction REsidual Sum of Squares). Then you can calculate a prediction R² replacing RSS by PRESS.

When you have much noise in your model, PRESS can be greater than TSS, so prediction R² can be negative (it's not a good case because even if R² is closed to 100%, it means than your model is closed to your observations but it is not usable to do predictions).

When analyzing a DOE, in your case a Box-Behnken design, so that the model you can estimate is a full quadratic model, very probably, several effects are not  statistically significant and then prediction R² is very low or negative. Eliminating one by one the unsignificant effects, you'll see R² decreasing slowly and prediction R² increasing.

So, what I think is a good model is a compromise between R², prediction R² and statistical significance of effects (after examination of residuals).

LOF is the part of RSS not due to repetitions of experiments. So, if you have few repetitions (classically 3 times the center point in a 3 factors BB design) you may have fortuitously very closed values for these experiments. A statistically significant LOF can be due to an outlyer you can identify looking standardized residuals, DFITS, ...

About R² or prediction R², I never give a limit for R² (or prediction R²) to say my model is good or not. I prefer a model with prediction R²=50% with only one effect to a model with prediction R²=90% and requiring 10 effects, each one explaining 5 to 10%.

Hope this will help you.

Regards,

Pierre Chagnon

Dear Nor Adilla Rashidi,

LOF (lack of fit) and R^2 (both R^2 and adjusted R^2) are used for testing the response of the model. From references that I read, if LOF was lower than 0.05, that means LOF was significant relative to the pure error. I think that maybe the LOF value of your study is lower than 0.05. The requested model is appropriate if LOF is higher than 0.05.

One more important to realize the role both R^2 and adjusted R^2 in the evaluation the significance of your suggested model, of course adjusted R^S is always lower than R^2, large differences between them demonstrate the insignificance of the model.

From the above reasons, if possible you can repeat some runs, especially for the factors more important. However, these values can not be enough to evaluate your results are good or not.

Regards,

Thanks for your suggestion. May I know what should be the difference between R-squared and adjusted R-squared? R-squared value of this model is about 0.8 and the adjusted R-squared is 0.6++.

Though R2 value is OK (> 80%), I wonder why I've obtained negative predicted R-squared.

Dear Nor,

Adj R2 takes into account the number of independent variables/regressors in your model whereas R2 does not.

As such if you are comparing the amount of variance explained by different models (with different number of independent var's) you should use the adjusted R2, otherwise it usually doesn't really matters.

R2 is just the fraction of the total variance that is explained by your model, so a R2=0.8 means that you model explains 80% of the observed variance.

The remaining 20% (that are not explained by your model) can be due to random variability (experimental variability) or to an effect that is not taken into account by your model (or a combination of both - which is almost always the case).

The significant LOF that you mention means that the amount of variance due to an effect not included in your model is significantly higher than the random experimental variability and you conclude that you are using a "wrong" model (not able to properly describe the experimental data) - therefore a "significant" part of those 20% of non-explained variance are due to LOF and a small part to random experimental variability (I'm simplifying a bit by ignoring the number of total observations, replicates, variables etc).

This significant LOF may be a "true LOF" or just an artifact due to the data, its precision and experimental design. Only looking at the data one may make a guess on what is really turning that LOF significant. The best and simplest way I to take a look at the residuals plot and check for a trend and/or pattern in the residuals. If you find one it’s almost sure that you’re facing a “true” significant lack-of-fit.

As you not disclosed any details on your experimental design (other than that you are using a Box-Behnken model) – no number of var’s/factors, number of replicates etc, etc - I’m not able to give you any additional help. If you are willing to do so I may take a look at your data and give some additional insights – just send me your data by email in a readable format (excel, CSV or similar).

Best, Luis

Dear Nor,

R² is, by definition, equal to (sum of squares explained by the model)/(total sum of squares). So, if you add an effect in your model, even if it's noise, R² will increase. So, R² is not necessary a good indicator for the quality of your model (you can build a model with R²=100% with only noise).

If we write: ESS=Explained Sum of Squares, TSS=Total Sum of Squares and RSS=Residual Sum of Squares then R²=ESS/TSS.

ANOVA tells us: TSS=ESS+RSS so: R²=(TSS-RSS)/TSS and then R²=1-RSS/TSS.

If you uses cross validation (usually leave one out), you can calculate PRESS (Prediction REsidual Sum of Squares). Then you can calculate a prediction R² replacing RSS by PRESS.

When you have much noise in your model, PRESS can be greater than TSS, so prediction R² can be negative (it's not a good case because even if R² is closed to 100%, it means than your model is closed to your observations but it is not usable to do predictions).

When analyzing a DOE, in your case a Box-Behnken design, so that the model you can estimate is a full quadratic model, very probably, several effects are not  statistically significant and then prediction R² is very low or negative. Eliminating one by one the unsignificant effects, you'll see R² decreasing slowly and prediction R² increasing.

So, what I think is a good model is a compromise between R², prediction R² and statistical significance of effects (after examination of residuals).

LOF is the part of RSS not due to repetitions of experiments. So, if you have few repetitions (classically 3 times the center point in a 3 factors BB design) you may have fortuitously very closed values for these experiments. A statistically significant LOF can be due to an outlyer you can identify looking standardized residuals, DFITS, ...

About R² or prediction R², I never give a limit for R² (or prediction R²) to say my model is good or not. I prefer a model with prediction R²=50% with only one effect to a model with prediction R²=90% and requiring 10 effects, each one explaining 5 to 10%.

Hope this will help you.

Regards,

Pierre Chagnon

Nor, Real data contain often surprise. Myself, I would fit a simplified model, possibly by other software tool, and analyse simple factors separately. Also, to depict all kinds of averages. And how is increasing the R-square, when You add factors to your models. Good luck

Anna

Dear Dr Pierre,

I have tried removed the insignificant factors, yet still obtained negative value for predicted R-squared.

Dear Nor,

At this point, it becomes difficult to propose you new routes to progress. The simplest would be to send me your data. Of course, you can hide the names of factors (x1, x2, ...) and responses (y1, y2, ...).

Regards,

Pierre

Dear Respected colleague,

In order to assess the overall regression model fit in supporting the research hypotheses. This is done by, firstly, examining the adjusted R squared (R2) to see the percentage of total variance of the dependent variables explained by the regression model. Whereas R2 tell us how much variation in the dependent variable is accounted for by the regression model, the adjusted value tells us how much variance in the dependent variable would be accounted for if the model had been derived from the population from which the sample was taken. Specifically, it reflects the goodness of fit of the model to the population taking into account the sample size and the number of predictors used. Researchers suggests that this value must be equal to or greater than 0.19.

I hope this answer your question,

Best Wishes,

Dear Nor,

I would first look at the actual RESULTS not only the statistics to try and understand what happens. Maybe your data just does not vary according to the generalized liner model, which only uses linaer or quadratic type dependancies between variable and data. This is for example almost always the case when people introduce time as a variant in a BoxBenkhen. Regression is not the be all and end all of model building!

Somebody suggested that you look at the residuals and I totally concur.