I have tried several common practise.
1. Moving all 5*5 and 7*7 conv filters to 3*3 or 1*1.
2. Try float16 instead of float32 (I used NVIDIA apex package).
Do your guys have any other suggestions?
Maybe you can paralyze the system.
I suggest you to improve the number of layers which is fully connected.
Dear Zhedong Zheng ,
For the purpose of saving GPU memory, you can modify CNN model. But it is application oriented task. I have some suggestions for you.
1. You can use layer reusing technique like DenseNet architecture. It saves training time and memory, both of them.
2. If you are working on image classification, then you can minimize the size of your input image.
3. Use small size of filters (2x2 or 3x3) to the convolutional layer.
4. In fully connected layer minimize the number of hidden layer and neuron numbers.
5. Use overlapping pooling layer (max or average), sometimes it works so efficiently. For further reference please see the paper of AlexNet model.
6. Data type changing (that means float32 to float16) can minimize the accuracy of CNN model but it is also helpful to save memory.
And finally check all of your functions in your code.
Thank you.
From the way this question is posed, I am assuming it is a GPU programming issue. based on this assumption I would inquire on why are you moving the blocks individually instead of just one big memcpy.
Thank you. All your suggestions are great.
I am running a generative model which demands about 14GB GPU memory. I am working on move this model to GTX1080 (with 8GB GPU memory limitation).
- Badges
- Science topic