Out-of-sample validation (validation of model performance on an independent set of data) is a must in physical modeling, data mining, etc., but in a traditional statistics course it is not taught and never (seldom?) practiced. I suspect that's connected with existence of a prescribed data model (which however is never tested either). Any thoughts on that?
.
the courses i know of do not ignore the train/test split (usually in the section Model validation) but "classical" statistics-oriented textbooks tend to favor resampling strategies instead of train/test split, which is consistent with the emphasis on the treatment of small or moderate size problems (and yes indeed, there are also intelligent lifeforms outside of big data !)
.
for instance
http://biostat.mc.vanderbilt.edu/tmp/course.pdf
.
In my opinion, it is based on the objective of the course, if it is introductory, then the main goal how to explain the idea of the models/methods. However, if the course is advanced and more practical then it is essential to add the out-of-sample validations.
Thank you Fabrice, good course. Note however that the chapter on validation ends with "How Do We Break Bad Habits? Insist on validation ...", which exactly makes my point.
Traditional statistics was developed for relatively small datasets (e.g. one sample t test developed by Gosset). These data sets do not allow for a (meaningfull) split in training/validation set or some sort of cross validation. Computationally this was also not feasible as a lot needed to be calculated by hand. A lot of attention in statistical modelling was paid to tests for significance of different models and/or combination of predictors which was the way (model-)validation was performed. With the advent of computers, other ideas emerged but the old flow of reasoning still exists and this is reflected in courses on these topics.
- Badges
- Science topic
- Similar topics
- Computer Science
- Data Mining