maybe it would help to additionally define a learning rate decay (please see link).

https://www.tensorflow.org/versions/r0.11/api_docs/python/train.html#exponential_decay

Try different network configurations.

Or, maybe you reached a local minimum, that's why the value of the cost function oscillates. Try to add a small random noise to avoid the local minima or try another method instead of gradient descent - stochastic gradient descent or use a Radial Basis Function neural network.

regards,

Bogdan

Well i try the two strategies the oscillations became more stable, but it never stop oscillating.

I find that the problem is when we use batch training and online training. When I use online training the loss always decrease (expected behavior).

For online training the code is:

regressor = tf.contrib.learn.DNNRegressor( feature_columns=None, optimizer=tf.train.GradientDescentOptimizer( learning_rate=learning_rate ),

hidden_units= [10, 10, 10], activation_fn=tf.nn.sigmoid, model_dir="/home/edwin/workspace/tensorFlowPy/eval")

regressor.fit(training_set, target_set, steps=10000) #Without the batch_size parameter

Maybe the accumulation of the deltas in batch training could cause this behavior,  do you have any ideas about that? Thank you for your comments. 

Sorry, I don't have a clear intuition here. Maybe of interest: in the linked tutorial for batch learning the "Bold Driver" learning rate adaption is introduced. But it's not directly available in TF as far as I can see, and I don't have any practical experience with it.

http://www.willamette.edu/~gorr/classes/cs449/momrate.html

Hola. Puede estar sobre-entrenando (overtraining).

is your  learning database not too complex?

The Database is not complex i try it with Boston database and I generate a database based in Sin(x) function. I find that this fluctuation is caused by the batch_size. If we use the whole training database in each epoch the model don't show this behavior. The noise generate by a small batch_size could be useful to escape from local minimum. After some tests i found that (1/4) of the training set could be used as batch_size without severe fluctuations (this might be tested in other sets). Bellow the evolution graphic is showed. I will appreciated any comments. Thank you.

Dear Edwin,

Although the behaviour shown by your model is rare, yet I think it's somehow explainable. You are right that stochastic gradient serves as some noise during learning and may help escape local minima. However, the samples for each class contained in successive batches may be somewhat inconsistent (so noisy) such that weights updates is erratic and therefore causes the value of your loss function to oscillate in the error-weight space. As you pointed out in your case, increasing the batch size exposes the neural network to more samples per class every iteration; and hence, there's lesser influence of highly noisy samples on weights update, and consequently the minimized loss experience little or no oscillate. i.e. consistent decrease in the loss. I hope this helps.