In my opinion we can you find the autoencoder in RNN architecture, I suggest if you can you use CNN architecture, maybe that can you give the good results without the overfitting.

I hope that be Claire for you. @ Anh Tram Nguyen

In my opinion we can you find the autoencoder in RNN architecture, I suggest if you can you use CNN architecture, maybe that can you give the good results without the overfitting.

I hope that be Claire for you. @ Anh Tram Nguyen

Could you please mention the model architecture that you used and the type of data used.

I don't think there's enough information in the question to give a straightforward answer, but here's something you can consider:

If these performance tests were done on the training data itself, then it may be reasonable that the training "performance" is decreasing with a larger dataset: it's likely that the smaller dataset was overfitting (i.e. high variance and low bias). In general, increasing the size of the training dataset should decrease both variance and bias in the test dataset. Additionally, increasing the complexity of the network may reduce this decrease in training performance.

Here's an interesting article you can have a look at: https://www.pnas.org/content/116/32/15849

Mohammad Khalid Pandit I used unsupervised autoencoder model to learn a normal dataset, then I use that trained model to detect anomalies on fraud dataset

Randall W Gladen Barodi Anass Thanks a lot, I'll look at that!

Probably you were over fitting your neural network with less data.

But since you're question is unclear, I mention few methods to avoid such problems:

1- Use CNN.

2- Instead of dropouts, use Batch Normalization layers.

3- Try different batch sizes.

4- Use Adam optimizer with lower learning rate.

5- Use transfer learning with known neural network like VGG16 and ResNet etc.