If I have k different training sets, I also have k different models with k different metrics (k AUCs, k True Positive Rates, etc). For metrics I consider the averages of k values, but which model do I select? If I have k inductive decision trees, what is the final tree?
The short answer: For your final model you need to train a new model using all the available data.
More detailed answer: When creating a model there are 3 basic questions that we need to answer: 1) Given a set of hyper-parameters (depth of the tree, number of neurons, regularization parameters, etc..), what is the model that optimizes the loss? 2) How do we find the best set of hyper-parameters? 3) What is the expected performance of the model trained the learning algorithm (which selects the best hyper-parameters)?
Answering each question requires different procedures. We use cross-validation to give an answer to the question #2. We try different sets of hyper parameters, and chose the ones with the highest average performance across all the folds.
Once that we have those "optimal" hyper-parameters, we can answer the question #1. We take all the available data and use only the best hyper parameters to train our final model.
If we want to get an estimation of the performance of that final model, then we need to do nested cross validation. In the first cross validation we create a temporal train set, and a test set. Then we further divide the temporal train set into the train set, and validation set. Then we use the validation set to determine the best parameters. After selecting the best parameters, train a model using the temporal train set and the measure the performance on the test set. Then we repeat this process 5 times...
If you don't need to set any hyper-parameters, the process is very similar. You use simple cross validation to get the estimation of the performance of your model. Then, for the final model, you simply train using all the available data.
Roberto Vega is correct, your final fitted model will be based on all available data. Your question is a common one, and it comes from a slight misinterpretation of the terminology, one that I have had in the past also. A model refers to the general structure. Model selection refers to selecting the best structure to represent your data, e.g. random forest vs. SVM. Cross-validation can help to compare the performance and robustness of the various models you are fitting. For your situation, you have already selected an inductive decision tree as your model and are looking at how different instances of that model fit different subsets to get an idea of how your model choice will reflect the overall population. The only thing this will tell you is if your model choice falls within a certain acceptable range of metrics, i.e. AUC, FPR, FDR, etc. That is the model validation part, not model selection.
In regard to which metric is best, that depends on what the model is used for.
In your case, what is the real-life cost of a false positive, compared to a false negative? Sometimes, you can answer both questions in dollars, so the decision is easy. Other times, some judgement is needed. In cancer screening, it is worse to pass a person with cancer than to refer for more tests someone who is healthy. The court system is based on the assumption that it is worse to imprison an innocent person than to free a guilty one.
- Badges
- Science topic
- Similar topics
- Organizational Psychology
- Training