You are asking a valuable question, Deng.

Too many researchers satisfy themselves that a model is good if it has acceptable R-squared and predictor variables are "significant". But prediction of an outcome based on data that were not used to create the model often gives a very different result.

The popular thing to do is to divide your data into two sets: a "training" set (say, 70%) and a "test" set (say, 30%). If you are using SPSS then there are procedures for randomly selecting cases and placing them into different data sets. Create your regression model using the training set and then test your model against the test set. R-squared of predicted values against actual values would be the appropriate statistic, but also popular are Mean-Squared Error (MSE) and Mean Absolute Deviation (MAD).

You may find that bootstrapping tools are also available in SPSS (I confess that I have moved over to R for this sort of analysis and forgotten a lot of SPSS). This means that you can get SPSS to run your analysis hundreds (or thousands) of times, and then give you averages of the model parameter estimates and average goodness-of-fit statistics on your hold-out (test) data.

You are asking a valuable question, Deng.

Too many researchers satisfy themselves that a model is good if it has acceptable R-squared and predictor variables are "significant". But prediction of an outcome based on data that were not used to create the model often gives a very different result.

The popular thing to do is to divide your data into two sets: a "training" set (say, 70%) and a "test" set (say, 30%). If you are using SPSS then there are procedures for randomly selecting cases and placing them into different data sets. Create your regression model using the training set and then test your model against the test set. R-squared of predicted values against actual values would be the appropriate statistic, but also popular are Mean-Squared Error (MSE) and Mean Absolute Deviation (MAD).

You may find that bootstrapping tools are also available in SPSS (I confess that I have moved over to R for this sort of analysis and forgotten a lot of SPSS). This means that you can get SPSS to run your analysis hundreds (or thousands) of times, and then give you averages of the model parameter estimates and average goodness-of-fit statistics on your hold-out (test) data.

In addition, you can try various forms of cross validation. The best reference books that I know of are given in the links below

http://biostat.mc.vanderbilt.edu/tmp/course.pdf

http://www-bcf.usc.edu/~gareth/ISL/

They also have software and examples. Best Wishes, DAvid Booth

Accuracy of models are often examined by validation through splitting the original data set into two parts in which one set is used for estimation and another set is used to see whether the predictions are accurate or not through various goodness of fit methods. You have not mentioned about multicolinearity and residual analysis in your model accuracy tests. These needs to be done for a good regression model.

Dear Xue Deng,

Apart from these very interesting and valuable recommendations by the other researchers, I want to recommend you to perform Applicability domain test and that is helpful to further examine the accuracy of your constructed model to predict the outcome.

Applicability domain (AD) is the “physicochemical, structural or biological space, knowledge or information on which the training set of the model has been developed”.

Thanks.