Well i am using DNNregressor of tensorflow to train a neural network with the gradient descent algorithm. This training is performed with a very low learning rate (learning_rate=0.001). But at the end of the training I observed that the optimization is not converging (the value of the cost function oscillate):
The code is:
regressor = tf.contrib.learn.DNNRegressor( feature_columns=feature_columns, optimizer=tf.train.GradientDescentOptimizer( learning_rate=learning_rate ),
hidden_units= [10,20,10], activation_fn=tf.nn.relu, model_dir="/home/edwin/workspace/tensorFlowPy/eval")
The ouput is:
INFO:tensorflow:Step 46701: loss = 0.00337208
INFO:tensorflow:Step 46801: loss = 0.00339828
INFO:tensorflow:Step 46901: loss = 0.000426068
INFO:tensorflow:Step 47001: loss = 0.000130174
INFO:tensorflow:Step 47101: loss = 2.59336e-05
INFO:tensorflow:Step 47201: loss = 0.0038842
INFO:tensorflow:Step 47301: loss = 0.000187496
INFO:tensorflow:Step 47401: loss = 0.00407278
INFO:tensorflow:Step 47501: loss = 0.00102294
INFO:tensorflow:Step 47601: loss = 0.00379042
INFO:tensorflow:Step 47701: loss = 0.000140399
INFO:tensorflow:Step 47801: loss = 0.00254074
INFO:tensorflow:Step 47901: loss = 0.00409194
Using gradient descend algorithm, in the worst case i expect that the training will stay in the best value of the cost function.
Thank you very much for your comments.
maybe it would help to additionally define a learning rate decay (please see link).
https://www.tensorflow.org/versions/r0.11/api_docs/python/train.html#exponential_decay
Or, maybe you reached a local minimum, that's why the value of the cost function oscillates. Try to add a small random noise to avoid the local minima or try another method instead of gradient descent - stochastic gradient descent or use a Radial Basis Function neural network.
regards,
Bogdan
Well i try the two strategies the oscillations became more stable, but it never stop oscillating.
I find that the problem is when we use batch training and online training. When I use online training the loss always decrease (expected behavior).
For online training the code is:
regressor = tf.contrib.learn.DNNRegressor( feature_columns=None, optimizer=tf.train.GradientDescentOptimizer( learning_rate=learning_rate ),
hidden_units= [10, 10, 10], activation_fn=tf.nn.sigmoid, model_dir="/home/edwin/workspace/tensorFlowPy/eval")
regressor.fit(training_set, target_set, steps=10000) #Without the batch_size parameter
Maybe the accumulation of the deltas in batch training could cause this behavior, do you have any ideas about that? Thank you for your comments.
Sorry, I don't have a clear intuition here. Maybe of interest: in the linked tutorial for batch learning the "Bold Driver" learning rate adaption is introduced. But it's not directly available in TF as far as I can see, and I don't have any practical experience with it.
http://www.willamette.edu/~gorr/classes/cs449/momrate.html
The Database is not complex i try it with Boston database and I generate a database based in Sin(x) function. I find that this fluctuation is caused by the batch_size. If we use the whole training database in each epoch the model don't show this behavior. The noise generate by a small batch_size could be useful to escape from local minimum. After some tests i found that (1/4) of the training set could be used as batch_size without severe fluctuations (this might be tested in other sets). Bellow the evolution graphic is showed. I will appreciated any comments. Thank you.
- Badges
- Science topic
- Similar topics
- Computer Science and Engineering
- Programming Languages
- Python