Assume that, after running some hyperparameter optimization technique based on the training data and possibly using cross-validation to guide the search, there are M best models avaliable to create an ensemble.
How can the performance of the method be assessed?
Some thoughts:
- The simplest way would be selecting just a set of M models (resulting of only one optimization run), create N replications (i.e. random train/test splits) and get the statistics. I believe this is a highly skewed approach, as it only consider the randomness of the ensemble creation, but not the randomness of the optimization technique.
- A second alternative would be creating N replications and, for each one of them, running the entire method (from hyperparameter optimization to ensemble creation). Then, we could look at the statistics for the N measures.
- The last alternative I can think of is creating N replications and, for each one of them, (1) run the optimization algorithm and (2) create K ensembles. In the end, we could extract statistics from all K*N measures.
Since I wasn't able to find good references on this specific issue, I hope you can help me :)
Please, cite published work that supports your answer.
Thank you.
Hola Victor
A common approach is to devise a contest for the M models, and select the best performing one. See (for example) Wichard's 2006 paper
Conference Paper Model Selection in an Ensemble Framework
Another approach is classifier combination. By combining the individual outputs from M models, we aim at a higher accuracy than that of the best individual classifier. Kuncheva's paper shows how to do this, and has useful citations to other work in this area.
Article A theoretical study on six classifier fusion strategies
Hello, dear Thomas.
Thank you for the answer.
I'm aware of Wichard's work, and the other paper didn't help me on the question.
The problem presented isn't how to build an ensemble of classifiers from the output of some optimization method. I already have done that.
I'd like to know what is the right way of testing my procedure to build the ensemble in terms of accuracy (or any other statistic of choice).
The three listed thoughts I shared will give the feeling of what I'm looking for.
Oi Vitor,
I personally always fit the ensembles as I would fit any model, with k-fold cross-validation or bootstrap, using the standard metrics (R2, MSE, Gini index, accuracy, Kappa, etc.). Do not worry about bias from optimizing the components of the ensemble - after all, they were cross-validated right? Try out with a toy example and compare performances.
Btw, I don't think selecting the single best model is ideal - that is not an ensemble! What you should definitely keep in check, is that the predictions of the M models you want to plug in into the ensemble have uncorrelated CV predictions (scatterplots will help). This is because models that yield similar predictions will not contribute much to the ensemble. Boa sorte!
Francisco
- Badges
- Science topic
- Similar topics
- Mathematics
- Statistics