I’m training some supervised machine learning algorithm to perform the prediction of a continuous variable.
I’m currently applying a nested cross-validation protocol (inner: LOOCV; outer: LOOCV; sample). It is a pilot and I have only 70 cases at the moment, while a test may come later if the pilot goes well.
I’m looking for a general strategy (aka applicable to any supervized technique, potentially also to an ensamble of them) to construct prediction intervals both for each current outer loop predictions as well as for future predictions of new subjects not included in the current sample.
Thank you!
Hi,
Have you used classification (as supervised method), or clustering to predict that intervals? In the case maybe consensus clustering could be a good method for partitioning task.
Using cross-validation depends on your data_set characteristics and not always best one.
I mean you may design an partitioning model of training by part of your data or multiple times learning (by clustering or classification), and then test part by assigning to predicted intervals.
It is a useful method for partitioning time series and prediction.
I hope it gives an idea!
Best,
Reza
Thank Zhongmin and Reza for your reply.
Zhongmin, yes, I would like to be free from assumptions both on the outcome and on the residuals, for the latter especially because I'm trying different techniques to see which can solve the task better. The only thing is that the outcome has theoretical min and max values.
Reza, thanks for your suggestion. Could you please share same references where I can find more info on the approach you propose?
Thank Zhongmin! I forgot to mention that I'm not working with time-series. The task is to predict for every patient how efficaciuos would be a potential treatment using baseline (aka before the treatment starts) information.
Dear Massimiliano,
I use predicting intervals in a spatio-temporal Event-Related Potential (ERP) data as well as Electroencephalogram (EEG), which recorded by many scalp electrodes from the brain (it could be continuous data in time domain), so in my experience I use clustering to find the intervals or predict them based on subjects brain activities. Here are some good reference for this concept:
Article A Tutorial on Data-Driven Methods for Statistically Assessin...
Article Principles of Topographic Analyses for Electrical Neuroimaging
Indeed, these are techniques for assigning time point to the right class. both these papers are using cross-validation to find that temporal intervals,
You may use different clustering method for partitioning, one of our recent work is based on cluster aggregation or cluster ensemble :
Conference Paper Cluster Aggregation for Analyzing Event-Related Potentials
I hope it helps or give you some idea,
Sincerely,
Reza
- Badges
- Science topic
- Similar topics
- Organizational Psychology
- Training