I'm training a neural network (E3D-LSTM) for spatio temporal video prediction. It's quite a big network that has millions of parameters. The model is training perfectly when using my 12 CPU cores, but when assigned to my NVIDIA GTX card, an OOM error stating that the "tensor size is too big to be assigned" is throwing up. I tried reducing the the batch size, but the error still persists. I'm surprised that CPU cores had no memory issue for such a big network, but GPU is not able to train the model. Can somebody throw some light on why this is happening and what can be done to fix it?. Please find attached the log-file for the complete error log.
Edit: The model is training on GPU for very low batch sizes, but not able to for reasonable batch sizes.
Hi Vishal,
I think that the problem is that this GPU only has 4 GB of memory, it looks like you need much more memory to train your network.
Best!
Try using Kaggle's free cloud computing infrastructure maybe you'll get even better results than using your local GPU.
The answer to why this happens is actually simple when you break it down. First, the CPU is not bound by GPU memory constraints. I have 32 GB DDR4 which the CPU has full unmitigated access to after acknowledging the OS needs to run as well, so leaving me with roughly 27-28BG give or take. My RTX 2060 card has a finite limit of 6 GB.
Now, most Nvidia cards max out at 11 GB and are fundamentally different when operations are taking place. For example, you have a huge level to render roughly 30 GB, but you don't have 30 GB so what happens is that it only renders what you are actively looking at frame by frame, draw distance, FOV, textures, etc... I have yet to enter AI in my degree at this point but what is more than likely happening with the training model is fully maxing your GPU memory quite quickly, then hitting a hard wall at its upper limit.
What you probably need to do is have the trainer load only smaller portions of data sequentially as it's needed and dump the rest to your slower SSD or RAM. Keep it moving in and out at all times to keep you from running into the hard limit with your GTX card. If I'm not correct most GTX cards have 4 GB in most cases depending on your budget. Another solution would be to get a GTX Titan with 21 GB of memory, but they are quite expensive but may be worth the investment in your case. Hope this helps.
- Badges
- Science topic
- Similar topics
- Organizational Psychology
- Training