The idea is that your validation sample is representative of the population. In that sense having a poor accuracy on the validation sample indicates that your machine learning model has not generalized beyond the training set. Continuing with several validation sets until one is tested with sufficient accuracies will lead to false positives.

How would this idea that "validation sample is representative of the population" be justified for a small sample? In biomedical studies, for example, both training and test are necessarily small.

The testing part of a machine learning model may refer to model evaluation while training or model evaluation once trained. I will explain the first, as the second is quite case specific. Say you are using data X_train to train some model. And assume that you have some additional data X_test that you keep on the side.

While training, people often report the test set accuracy (accuracy of model on X_test) and the train set accuracy (accuracy of model on X_train). If these too diverge while training, then the model is said to be over-fitting the data. That is, it is too specific to X_train, and does not generalise well to data it has not seen.

Coming to your second question, as in any experiment design, those collecting data need to make sure that the data-set is rich enough. In machine learning language, we would say it captures enough variability in the features being examined. Once a model is trained, it is still important to validate it against data coming from outside X_test and X_train, and assimilate new information by incorporating examples that cause the model to give wrong predictions. This is akin to learning in human beings.

I hope this helps!

I never asked about training.

You never answered my only question about the statistical meaning of testing.