I have about 10k subjects with each having 1000 observations at fixed time points(Y). I have built a model and made predictions for each subject (\hat{Y}). Now I would like to apply Deep Learning to reduce the gap between Y and \hat{Y} to improve the prediction performance. What I'm thinking is to use \hat{Y} to predict residuals Y-\hat{Y} with Deep Learning.
Basically, for Deep Learning, the input is 1000 dimensional \hat{Y}, and output is 1000 dimensional residual for each subject.
I have been searching articles and tried to find how to do it, but no clue. Does anyone have any suggestions?
Dear Gang,
Before looking at deep learning, it is worth determining what is causing the prediction error in your system. Oftentimes, simple improvement in dimensionality reduction with principal component analysis can help reduce variance. For this application, you can also implement a feed forward neural network in python as well that will automatically learn features from linear combinations of your observations.
Dear Gang Yang :
There are extensions of recursive partitioning and trees for longitudinal and clustered data. They essentially include a mixed model element into the algorithm. Here are a few papers that explicitly aim to create a machine learning and deep learning models from longitudinal/panel data:
Article Mixed Effect Machine Learning: a framework for predicting lo...
- Zhong et al (2019). Machine learning discovery of longitudinal patterns of depression and suicidal ideation
Article Machine learning discovery of longitudinal patterns of depre...
- https://papers.ssrn.com/sol3/papers.cfm?abstract_id=3159577
Best Regards;
Ehtisham
- Badges
- Science topic
- Similar topics
- Phytochemistry
- Extracts