Hello,
In these days i'm trying to train a deep neural network with a matrix with 14382 variables x 671 cases, from IV spectral data. I'm using R with Keras package (compile and execution commands bellow). The training set error is low, in some cases, decimal, whereas the error of validation set is very high and this two curves do not encounter at the computation end nor the validation error minimizes - it remains basically constant.
what i have to do to minimize the validation error in this ML challenge?
commands:
modelo%
layer_dense(units=1250,activation='relu',input_shape=c(14381)) %>%
layer_dense(units=630,activation='relu') %>%
layer_dense(units=630,activation='relu') %>%
layer_dense(units=410,activation='relu') %>%
layer_dense(units=1)
modelo %>% compile(loss='mse',optimizer='adam',metrics='mae')
mymodel%
fit(training,trainingtarget,epochs=1500,batch_size=250,validation_split=0.2)
you can use regularization techniques to improve the performance and solve the over-fitting problem ( such as dropout, activity_regularizer, kernel_regularizer.... ...etc) . See keras python or R documentation.
you can have a look on this code as well in R language
Code Deep sparse denoisig with DNN (R code )
This problem occurs basically due to overfitting. You can insert dropout layers inbetween and then check the result
Hello Lucas Almir Cavalcante Minho,
First of all, I would try to play with regularization as said before. Classical techniques are Dropout, L2, and L1. I would suggest you to play with dropout and L2 as L1 is less used.
Also, you can try using batch norm avec each dense layers as it has some regularization properties. Note that batch normalization has different properties with different batch sizes. If you add batch norm, the lower your batch size is, the higher the regularization effects will be.
You can also try another loss, recently I had a network with a similar issue. Switching from MSE to Logcosh helped me.
One last thing : you have a LOT of features. You may be facing the curse of dimensionality : try either feature selection, or dimensionality reduction techniques (PCA for a linear reduction, Kernel PCA or an Autoencoder for non linear dimensionality reduction).
You can also try data augmentation, like SMOTE, or adding noise (ONLY to your training set), but training with noise is the same thing as the Tikhonov Regularization (L2 Reg).
Hope you'll find a solution!
So... Many thanks for the excellent ideas. Muna Al-Hawawreh , Saloni Bhatia , Mohammed Deeb and Clément Poiret , i have used dropout regularization techinique and the overfitting issue have been controled, but the validation error is still high and when i have to do a prediction with the test set, the results is very divergent. What i have to do to improve the predition capacity of my deep neural networks? And Clément Poiret , i used a random forest algorithm to make feature importance analysis and selected 14 variables, but the problem is still persistent (but with a smaller error).
Mohammed Deeb, Do you know any transfer learning or pretrained model arcticles or algorithms to apply to this regression problem? These techiniques are frequently used for image recognition, how to apply these techiniques to my ML problem?
Lucas Almir Cavalcante Minho
Have you normalized your data before using deep learning (e.g., Min-Max Scaler)
Have you tried to shuffle your data during training ? (Shuffle=TRUE).
Maybe this will help as it is hard to know what is the problem without seeing the data and the code...
Best of luck
The training and validation errors should not have such a sharp difference. This is due to the over-fitting. To solve this issue you can test the following:
- normalize both input and output data between 0 and 1 since you use RELU activation function
- use smaller number of filters or apply dropout regularization to tune the network
- apply early stop regularization
- Badges
- Science topic
- Similar topics
- Organizational Psychology
- Training