I have trained a single layer neural network with 60 neurons as input layer, 30 neurons as hidden layer, 60 neuron as output layer.
When I use matlab neural network toolbox, training process takes about 5 hours even by using parallel computing (3 cpus).
I then trained such NN in tensorflow and the training process took only 1 min.
I can not find the reason for such difference. both NN have same architecture and trained with same data. The only difference is the learning algorithm which is Levenberg–Marquardt in Matlab and Back propagation in Tensorflow.
Did you have same experience? or/and do you know what is the reason for such difference?
I haven't used MATLAB NN Toolbox myself, however, Levenberg–Marquardt is bound to be slower than gradient descent. LM requires computing Hessians (Jacobians) and then inverting them. I am not sure about the implementation of LM in MATLAB, however, if it does not use any Hessian-free approach, then the implementation is bound to be slow.
You have at least ( (size_of_input+1)*60 + 61*30 + 31*60 + 61*size_of_output) parameters. Computing Jacobian each time would require O(N^2) calculations and taking inverse would be O(N^3). On the other hand, gradient descent is O(N).
However, this is just my guess as I am not aware of implementation in MATLAB. Try gradient descent in MATLAB, and that shouldn't be as slow presumably.
Cheers,
Mayank
Depends a lot on the actual implementation. As far as I'm aware Matlab does adds some layers on top of the direct processing, while Tensorflow is directly on top of the OS (depending on the OS you have)
Also, the NN Toolbox for Matlab is not really up to dat, so I would think the TF implementation is better suited for some tasks.
I for one would check how your cores are being used when you run both jobs.
Thank you Leon, you are right. Also Mayank advice was helpful.
I trained the NN with matlab using Gradient Descent and the training time improved so much.
But still TF is much faster in training process which is related to what you mentioned.
- Badges
- Science topic