I have two methods for doing Monte Carlo simulations. With both of them I have run serveral simulations and got the mean and variance of their results. I would like to determine whether both methods are equivalent. How can I compare them since hypothesis tests like Student's t test only determine that there is not strong evidence to refute the null hypothesis?
Dear Carmen Yago ,
null-hyposesis significance tests (NHST) such as t-tests, are only able to provide evidence against the null hypothesis. Therefore, NHST can not be interpreted as evidence against the presence of an effect, which is equivalent to an interpretation in favor of the null hypothesis.
I think possible solutions might come from the area of equivalence testing, i.e., evidence that two values are practically equivalent.
This can be done by either using frequentist approaches (Lakens, D. (2017). Equivalence Tests: A Practical Primer for t Tests, Correlations, and Meta-Analyses. Social Psychological and Personality Science, 8(4), 355–362. https://doi.org/10/gbf8nt ) or Bayesian approaches (Kruschke, J. K., & Liddell, T. M. (2018). The Bayesian New Statistics: Hypothesis testing, estimation, meta-analysis, and power analysis from a Bayesian perspective. Psychonomic Bulletin & Review, 25(1), 178–206. https://doi.org/10/gc3gmn).
Also from the Bayesian perspective comes the idea of the Bayes Factor that expresses the likeliness that a hypothesis (simulation results from both MC-simulations are equal) is true given the data (Lakens, D., McLatchie, N., Isager, P. M., Scheel, A. M., & Dienes, Z. (2018). Improving Inferences About Null Effects With Bayes Factors and Equivalence Tests. The Journals of Gerontology: Series B. https://doi.org/10/gdk2z5). Not to be confounded with the likeliness of a Value to occur given a certain hypothesis as in NHST.
Bests,
Sven .
Dear Carmen Yago ,
null-hyposesis significance tests (NHST) such as t-tests, are only able to provide evidence against the null hypothesis. Therefore, NHST can not be interpreted as evidence against the presence of an effect, which is equivalent to an interpretation in favor of the null hypothesis.
I think possible solutions might come from the area of equivalence testing, i.e., evidence that two values are practically equivalent.
This can be done by either using frequentist approaches (Lakens, D. (2017). Equivalence Tests: A Practical Primer for t Tests, Correlations, and Meta-Analyses. Social Psychological and Personality Science, 8(4), 355–362. https://doi.org/10/gbf8nt ) or Bayesian approaches (Kruschke, J. K., & Liddell, T. M. (2018). The Bayesian New Statistics: Hypothesis testing, estimation, meta-analysis, and power analysis from a Bayesian perspective. Psychonomic Bulletin & Review, 25(1), 178–206. https://doi.org/10/gc3gmn).
Also from the Bayesian perspective comes the idea of the Bayes Factor that expresses the likeliness that a hypothesis (simulation results from both MC-simulations are equal) is true given the data (Lakens, D., McLatchie, N., Isager, P. M., Scheel, A. M., & Dienes, Z. (2018). Improving Inferences About Null Effects With Bayes Factors and Equivalence Tests. The Journals of Gerontology: Series B. https://doi.org/10/gdk2z5). Not to be confounded with the likeliness of a Value to occur given a certain hypothesis as in NHST.
Bests,
Sven .
Suggest you look at model validation sections in: Clinical Prediction Models: A Practical Approach to Development, Validation, and Updating | Ewout W. Steyerberg | download (b-ok.cc) Best, D. Booth
There are a set of tests for Method comparison. I did a review paper on this. Have a look at the attached reprint...
An additional idea to the other proposals: Since simulations can in principle be repeated as often as desired and are only limited by the computing power, the statistical power of the tests can be increased as wished. This somehow makes inferential testing superfluous, at least if you have a large number of repetitions. Any effect will be significant then. In this case, e.g. with 10 000 repetitions, it may be advisable to skip statistical tests altogether and only provide measurements of the effect size to identify effects, which are really noticeable or relevant.
- Badges
- Science topic
- Similar topics
- Biological Science
- Kinesiology
- Running