May be u r partially right.

.

the courses i know of do not ignore the train/test split (usually in the section Model validation) but "classical" statistics-oriented textbooks tend to favor resampling strategies instead of train/test split, which is consistent with the emphasis on the treatment of small or moderate size problems (and yes indeed, there are also intelligent lifeforms outside of big data !)

.

for instance

http://biostat.mc.vanderbilt.edu/tmp/course.pdf

.

In my opinion, it is based on the objective of the course, if it is introductory, then the main goal how to explain the idea of the models/methods. However, if the course is advanced and more practical then it is essential to add the out-of-sample validations.

Thank you Fabrice, good course. Note however that the chapter on validation ends with "How Do We Break Bad Habits? Insist on validation ...", which exactly makes my point.

Traditional statistics was developed for relatively small datasets (e.g. one sample t test developed by Gosset). These data sets do not allow for a (meaningfull) split in training/validation set or some sort of cross validation. Computationally this was also not feasible as a lot needed to be calculated by hand. A lot of attention in statistical modelling was paid to tests for significance of different models and/or combination of predictors which was the way (model-)validation was performed. With the advent of computers, other ideas emerged but the old flow of reasoning still exists and this is reflected in courses on these topics.

Most non-statisticians do not get beyond "Introduction to Statistics." This includes the people doing the research and the politicians who direct funding. So we get experiments that are funded with 5 to 10 replicates. This works because I need to finish one project every 4 months to maintain a publication rate of 3 papers/year. So I get a grant for $250,000. I run the experiment and the money is mostly gone as is the time. To do a validation study I will need more funding and more time. The granting agency claims that the work was already done, or the reviewers claim (about the validation study) that the results were already published in the original work and the work is no longer novel. I could try to get them published together, but then the university claims that I am not being productive enough because I only had one manuscript this year and I am promptly fired.

One of the things I remember out of introductory statistics was the teacher happily showing the class that a sample size of 4 or 5 was sufficient. Later, my Ph.D. advisor ran field plots with a target of 4 replicates, plus two replicates in case something bad happened. Essentially using two replicates as insurance.

This is the context within which statistical decisions are made. Mostly people think of statistics as a service like any other piece of lab equipment, and one only needs that service after all the data are collected. Unfortunately, that is too late. It was already too late when I submitted a grant for 450,000 to run an experiment with 20 replicates and my competitor submitted the same grant for $250,000 and 5 replicates. The 5 replicate study becomes definitive once published and it takes great effort to get the paradigm overturned.

Skill level: I can run a t-test all by myself and it will take me 5 minutes or less. Writing a program to validate original results is more involved.

Sampling: Am I better off running two experiments with a sample size of 10 versus one experiment with a sample size of 20? The sampling accuracy is much better with a single larger sample. I suppose that what you need is to do one experiment with sufficient sample size to have reasonable power, and then run a validation test with the same sample size. Unfortunately, before the experiment is run there is no real way to "justify" a large sample and afterwards there is insufficient data to refute the veracity of the small sample.