Usually, the common ways to improve a deep learning model are the followings.

1. Adding layers

2. Increasing the number of hidden units

3. Changing the activation function

4. Changing the optimization function

5. Changing the learning rate

I would like you to tune your model by following the above steps one by one, so that you can track down the performance. Regarding the learning rate, you can start with a very low learning rate of 0.001, and then increasing it gradually three times, (e.g., 0.003, 0.01, 0.03, 0.1, 0.3). Data augmentation can be another very useful way to improve the models performance.

S M Asiful Huda Thanks so much for the answer, unfortunately I don't to change/add too much layers or hidden units to the base VGG16 model. Our study focuses more on the comparison on different models and their different iterations/versions.

To enhance the performance of your VGG16 model during training and validation, you can start by applying data augmentation techniques to increase dataset diversity and reduce overfitting. It's crucial to ensure the dataset's cleanliness, correct labeling, and appropriate division into training, validation, and test subsets. Experiment with different learning rates and optimizers, and consider using learning rate schedulers if necessary. Employ regularization methods like dropout and L2 regularization to tackle overfitting issues. Keep a close eye on the training process, implement early stopping, and adjust the batch size as needed. You might also want to explore alternative model architectures or smaller models that could better suit your dataset. Lastly, make sure your hardware resources are utilized effectively, and explore ensemble methods to potentially enhance model performance. These strategies should help you overcome the low accuracy challenge with your VGG16 model.