The concept behind using autoencoders is to regenerate or replicate the same input. I have as input a 6By6 matrix that represent a grid layout image. I used as training and testing data for Disentangled variational autoencoders. However , the model couldn't regenerate the same input.
I am wondering the logical reason behind not using a single matrix for training the model ?
Hi in order to build a deep learning model that fit your training dataset you need to have first a training dataset with enough data making the model able to learn, discover and extract useful features. Till now all deep learning models depends on this fact. Because the model needs more observed data to fit and learn the data distribution.
Dear Jaoua,
In principle, a variational autoencoder has the inference part (encoder) which tries to find the latent variable/s "z" (setting a prior such as "z" being a Gaussian with mean mu and variance sigma) given your data X; and the model part (decoder) which allow us to sample from the given Gaussian and get the likelihood of p(X|z). During the optimization process of this model, parameters of the network have to be learned from data as usual and you must have enough data to do this, i.e. have as many instances as possible in your dataset X.
Daniel Urda Dairi Abdelkader
I have done a small experiment with my data. I have made 10 different examples of grid layouts and then replicated them 1000 times. This is my input data for my VAE model. Then , for testing the model, I give it 10 grid layouts but they were not regenerated the same. How do you explain this behavior please ? Should I increase more the data by replicating 10000 times ?
Hey,
I don't know if simply replicating the data helps. Bringing a little variety into the samples by turning them around, mirroring, zooming etc. might help.
What is your training accuracy and what layer setup do you use?
Hi Jaoua,
Although you are replicating samples 1000 times to train your VAE, you are doing it based on just 10 different instances having very little variability in the input space. You will need more different grid layouts if you want your model to perform well in new future data.
To get the intuition, have a look at the polynomial regression example explained in chapter 1 of Bishop’s book:
http://users.isr.ist.utl.pt/~wurmd/Livros/school/Bishop%20-%20Pattern%20Recognition%20And%20Machine%20Learning%20-%20Springer%20%202006.pdf
Hi,
replicating the same data will not help you ! the VAE try to learn the underlining data distribution of the input in order to build your model which is at the end a weights matrix (W) with optionally bias vector (b) depending on the depth of your VAE. so after training process you'll have a constant (W,b) , to encode X : Z = sigmoid(X.W + b) and to decode use W transpose instead of W. for Each X you'll have Z. si it's normal to have different code depending on the input.
Yussuf Kassem As setup, the encoder contains 2 dense layers. On dense layer with 15 as intermediate dimension and another dense layer with 10 as encoding dimension. The opposite way applies for the decoder. As activation function, I am using the RELU function.
For the setup of 1000 replicated 10 different examples, I get as final training and validation loss : 0.027
Daniel Urda It's true that I am giving a data with less variance in its but for testing , I also give the same training data. But still, it generates the same input.
Does this include that my model did not learn the right way from those 1000 replicated data ?
Hey,
the validation set has to be independent from the training set. Does your validation set come from the 1000 replica? In having ~1000 replicated grids that you already trained with your validation set then would be known to the model, which explains the low loss in the case of validation. I assume that the inability of recreating the input also has to do with your setup and data variety. Maybe you should consider convolutional autoencoders to learn the smaller parts of a grid. Also having more than 10 basic grids will enhance your model. Deep neural networks usually need way more data to work their magic. You should use your 10 grids and create new data from them by turning, mirroring, zooming etc. That way your model has more input to learn from.
Yussuf Kassem
Thank you for your answer, I will try with the convolutional autoencoders as well.
In fact, I tried with just the autoencoders and it succeded to regenerate the given inputs.
So besides the fact that my data is not that variant, why do you think that the autoencoders succeded in regenerating while the variational autoencoders didn't ?
Hey,
I think it has to do with the fact that the variational autoencoder learns the underlying distribution while the autoencoder learns a mapping through compression and decompression. When you train an autoencoder it is fairly easy to recreate your input. Autoencoders are therefore good for tasks like denoising images or signals. The variational autoencoder however is a generative model. By learning the underlying distribution you will be able to create things that have not been seen from the model yet (e.g. new faces or in your case not witnessed grids). That makes me believe that the distribution in the case of your variational autoencoder wasn't captured by your model. And the reason for that might be the small input space.
- Badges
- Science topic
- Similar topics
- Computer Science
- Images