I have a multivariate time-series forecasting problem which I am trying to solve using LSTM models. I have transformed my data such that it looks back last 20 timestamps to reconstruct the records at current timestamp, using deep LSTM model. Following is the model architecture :
model3 = Sequential()
model3.add(LSTM(1000,input_shape=(train_X.shape[1], train_X.shape[2]), return_sequences = True))
model3.add(Dropout(0.2))
model3.add(LSTM(500, activation = 'relu', return_sequences = True))
model3.add(Dropout(0.2))
model3.add(LSTM(250, activation = 'relu', return_sequences = False))
model3.add(Dropout(0.2))
model3.add(Dense(train_y.shape[1]))
model3.compile(loss='mse', optimizer='adam', metrics = ['accuracy'])
model3.summary()
history3 = model3.fit(train_X, train_y, epochs=100, batch_size=36, validation_data=(test_X, test_y), verbose = 2, shuffle= False)
Attached is the graph of the neural network output. The 'validation loss' metrics from the test data has been oscillating a lot after epochs but not really decreasing.
Can anyone explain how to interpret the graph ? Also, what could be the potential ways to ensure that the model does get improved with every new epoch ?
The graph shows the overfitting during model training. You can check your training history, does accuracy show peak performance and then show a decrease during training? First, you can add more data to train the model and make sure the dataset has divided into data train, data evaluation, data test with properly based on the practice.
Asep Fajar Firmansyah. Thanks for answering my question. The behavior here is a bit strange. I see that accuracy of validation data is better in every epoch as compared to training but at the same time, loss value is much higher in test than the train. I just updated my question with additional details of the model summary and values from each epoch. Maybe, if you could take a look and interpret what is wrong here ?
Actually the graph doesn't tell us the entire story, It looks like the validation loss is oscillating a lot! But in truth it appears that way b/c you y-axis is scaled from 0 to 0.12, which is a very small margin. Make this scale bigger and then you will see the validation loss is stuck at somewhere at 0.05. Of course these mild oscillations will naturally occur (that's a different discussion point).
In your case, it seems the model is overfitting a little, but just the loss function is not enough. Check the difference in training and validation accuracy and then it will be clearer.
To overcome overfitting, there are many things you can do
(i) Add more regularization
(ii) Data augmentation
(iii) Increase more data samples naturally
(iv) Reduce hidden layers
etc.
How much data are you putting through your pipeline?
I ask this because your model may not be training all the layers. Creating just a more costly machine to run.
Why are you using RELU as the activation function?
When using RNN(LSTM) models you are looking at data that is adjusted according to the data that was priorly passed. You should keep it tanh or sigmoid.
Adjusting batch size and the loss function will also help with the oscillation.
- Badges
- Science topic