Hi Mayur,

there are many ways to validate or verify the quality of predictions. In meteorology there is even a sub-topic called forecast verification to study such issues: http://www.cawcr.gov.au/projects/verification/

Many of the scores presented are also used in other fields like remote sensing, machine learning, etc. Beware that some time they are given different names!

For your specific task I would fit the model parameters using only  a subset of the dataset or cross-validation techniques. I would then try to predict the last part of the dataset without parameter recalibration to assess the real prediction skill of the model.

Hi  Mayur;

There are many statistical tools for model validation, but the primary tool for most process modeling applications is graphical residual analysis. Different types of plots of the residuals (see definition below) from a fitted model provide information on the adequacy of different aspects of the model. Numerical methods for model validation, such as the R2 statistic, are also useful, but usually to a lesser degree than graphical methods.

Unfortunately, a high R2 (coefficient of determination) does not guarantee that the model fits the data well, because as Anscombe's quartet shows, a high R2 can occur in the presence of misspecification of the functional form of a relationship or in the presence of outliers that distort the true relationship.

One problem with the R2 as a measure of model validity is that it can always be increased by adding more variables into the model, except in the unlikely event that the additional variables are exactly uncorrelated with the dependent variable in the data sample being used. To avoid such spurious increases of the R2, one can instead use the adjusted R2, which penalizes the use of additional explanatory variables in accordance with the amount that they are likely to spuriously increase the R2.

See this URL

http://en.wikipedia.org/wiki/Regression_model_validation

Also you can use Out-of-sample evaluation which is the process of assessing how the results of a statistical analysis will generalize to an independent data set. If the model has been estimated over some, but not all, of the available data, then the model using the estimated parameters can be used to predict the held-back data. If, for example, the out-of-sample mean squared error, also known as the mean squared prediction error, is substantially higher than the in-sample mean square error, this is a sign of deficiency in the model. Prof. kleijnen clarifies many methods of validation for simulation models. This URL will be so usefull for your study.

http://papers.ssrn.com/sol3/papers.cfm?abstract_id=843

Best Regards.

Caution: proceed carefully when "validating" any model for predictive purposes.

Many unrealistic expectations and unscientific approaches plague this field.

The scientific objective of model validation is to determine the degree of model accuracy for specific intended uses.  In other words: how accurately will the model represent my specific predictions?  That degree of accuracy may not be representative of any other predictions using that model.

Why is that the objective of model validation?  The answer lies in the fact that all models are approximations or abstractions of reality; all are "imperfect" (i.e., "all models are wrong..."), so the scientific objective is, in effect, to determine how imperfect.  Some known degree of imperfection should be acceptable for any intended prediction, but the consequence of some predictions will drive much stricter accuracy requirements.

Common unrealistic expectations regarding model validation include:

- software V&V contributes to model validation. [Software V&V contributes only to model verification.  This is an example of a very unfortunate confusion of terminology.]

- model accuracy as measured against real world data will represent the accuracy of a subsequent prediction.  [This is false especially if the model is used to extrapolate beyond the real world data.  Extrapolation introduces additional variabilities and uncertainties that degrade the predictive power of models.]

- lacking real-world data, model validity can be supported by expert opinion of the plausibility of model results.  [This is a dangerous form of pseudo-science.  Plausible model results may justify the expense of efforts to determine scientific accuracy via comparison with real-world data, and the generation of new validation-quality real-world data.]

- model-to-model comparisons contribute to validation.  [This is another form of pseudo-science.]

- lacking a scientific assessment of predictive accuracy, computational model results are preferable to a lack of model results, and must be adequate for trend analysis.  [Self-deceptive pseudo-science; models have no inherent validity - even for trends.]

Hi Mayur,

regression model validation is the process of deciding whether the numerical results quantifying hypothesized relationships between variables, obtained from regression analysis, are in fact acceptable as descriptions of the data. The validation process can involve analyzing the goodness of fit of the regression, analyzing whether the regression residuals are random, and checking whether the model's predictive performance deteriorates substantially when applied to data that were not used in model estimation.

Hi,

Your question:  What are the different ways of performing validation of prediction model?  This question is very much interesting one.  Kindly see the below points.  Its very much useful to you in your research or study.

Validation of predictions from regression models is of paramount importance one.  There are three types of validations, they are as following:

1.  Apparent: The performance on sample used to develop model as per the protocol of the respective study.  In this apparent method: It is very easy to calculate and the results in optimistic performance estimates.

Tutorials in Biostatistics: Statistical Methods in Clinical Studies, Volume 1

Kindly see the link and buy the book:

http://onlinelibrary.wiley.com/doi/10.1002/0470023678.ch2b%28i%29/summary 

2.  Internal: The performance on population underlying the sample of the particular study.  It is more difficult to calculate one ad the test model in new data, random from underlying population.

Refer:  http://www.ncbi.nlm.nih.gov/pubmed/11470385 

3.  External: The performance on related but slightly different population in the study.  It is the moderately easy to calculate when new data are available in the prediction model.  The test model in new data, different from development population.

Refer:  http://clincancerres.aacrjournals.org/content/18/3/815.long 

Hello sir u can use the regression analysis for the priodiction. https://www.researchgate.net/profile/Rohit_Sharma57/publication/273146067_Optimization_of_Machining_Parameters_to_Achieve_Nano-level_Surface_Quality_on_Oxygen_Free_High_Conducting_Copper_Using_Diamond_Turning/links/5565613308ae89e758fd9fb4.pdf

Chapter Optimization of Machining Parameters to Achieve Nano-level S...

There are a number of ways to validate models, and what you use will depend on your purpose and the nature of the data being modeled as well as the validity of the "observations" here are some suggested papers:

Article On the Validation of Model

Article Statistics for the Evaluation and Comparison of Models

Article Assessment of three dimensionless measures of model performance