Dear all,
I would like to compare the performance of some robust methods for estimating regression meta-model ( a regression model for simulation model). What is your suggestion of validation technique for validating a simulation model with the estimated regression meta-model instead of cross validation & bootstrap technique, which will perform better?. Thanks in advanced
Best Wishes
Huda
Hi, Huda
Can you explain the reason why you exclude cross validation?
Recently, I upload the detail of the k-fold cross validation. I think it is useful for the regression in order to the discriminant analysis.
Verification is concerned with building the model right. It is utilized in the comparison of the conceptual model to the computer representation that implements that conception. Validation is concerned with building the right model. It is utilized to determine that a model is an accurate representation of the real system.
Many commonsense suggestions can be given for use in the verification process.
1. Have the code checked by someone other than the programmer.
2. Make a flow diagram which includes each logically possible action a system can take when an event occurs, and follow the model logic for each action for each event type.
3. Closely examine the model output for reasonableness under a variety of settings of the input parameters. Have the code print out a wide variety of output statistics.
4. Have the computerized model print the input parameters at the end of the simulation, to be sure that these parameter values have not been changed inadvertently.
5. Make the computer code as self-documenting as possible. Give a precise definition of every variable used, and a general description of the purpose of each major section of code.
These suggestions are basically the same ones any programmer would follow when debugging a computer program.
Important link
http://in.mathworks.com/solutions/verification-validation/
Hi Shuichi,
Thank you very much for your response. I want to compare the performance of four robust methods for estimating regression meta-models based on monte-carlo simulation. One of these methods is OLS based on re-sampling in linear regression (Bootstrap Method in estimating regression coefficients). Now, I have been written the simulation program to compare these four methods therefore, using cross validation will be so difficult.
Best Regards
Huda
Dear Dr Huda
The Validation technique was created by Naomi Feil 30 years ago. It is a tested method of communication with disoriented elderly that helps reduce stress, and enhance dignity and happiness. It's all about accepting the feelings of another person and reaching out to them with empathy
Validation is an automatic computer check to ensure that the data entered is sensible and reasonable. It does not check the accuracy of data.
Hi Aly,
Thanks a lot for your response, but I meant validation for simulation models .
prof. Jack P.C. Kleijnen defined validation as "determining whether the conceptual simulation model is an accurate representation of the real system (also see the classic textbook on simulation, Law and Kelton 1991)".
http://citeseerx.ist.psu.edu/viewdoc/download?doi=10.1.1.105.9004&rep=rep1&type=pdf
Also we can use validation techniques for determining whether the regression metamodel is an accurate representation of the simulation models .
Hence I would like to study the validation using statistical tools for the second purpose.
Thank you
Huda
Progressive validation is a technique which allows you to use almost
2 of the data in a holdout set for training purposes while still providing the same guarantee as the holdout bound. It first appeared in [3] and is discussed in a more refined and detailed form here.
http://www.comp.nus.edu.sg/~teoym/cs6205/9900/L4/
- Badges
- Science method