As it is described and proposed in some references such as [1], 5x2 cross-validation is a suitable approach for comparing two algorithms and we use it in our work. But, if we want to optimize the hyper-parameters of an algorithm for example MLP learning, (decide its controllable factors, e.g. the number of hidden layers or hidden neurons) are we forced to use the same method of 5x2 cross-validation that we use for comparing our algorithm with others. Or we can use 30-replications of the train-test method?
[1] Dietterich, T. G. 1998. Approximate Statistical Tests for Comparing Supervised
Classification Learning Algorithm, Neural Computation 10: 1895–1923.
Have you considered to letting a meta-heuristic process guiding you?
It may point out in a direction, it has not been thought yet.
Dear Patricia,
My algorithm is evolutionary. We use EA to evolve ANN by a new genotype encoding and I think that there must be some easier and more stable experiment design for analyzing our algorithm (consists of hyper-parameter optimization) than 5x2 cross-validation.
Also, in some papers from 2010, 30 run of train-test is used for this process while they used 5x2 C-V for comparing their algorithms against others. Unfortunately, they have not justified it.
That is interesting. I am evolving evolutionary algorithms. Lattices have been used in the to identify some patterns in the optimised algorithms. I have yet to apply them though.
Patricia
- Badges
- Science topic