In my opinion, it is common to get such differences between 5-9 neurons.

The initial value and the specialty in the problem can make the error value fluctuate with different hidden size.

In fact, it  is more confusing  to see the increasing error with 17 neurons if you get the result by averaging lots of runs to eliminating effects of initial values.

If at every run of your ANN the weights between the layers are initialized at random values that may be one of the explanations (as the fluctuation is so small). ¿Do you know how they are initialized?

Also there may be more possible causes like the number of training epochs, the input data type...

You can read my paper :

https://www.researchgate.net/publication/272481612_An_Intelligent_Control_Technique_for_Dynamic_Optimization_of_Temperature_during_Fruit_Storage_Process

Article An Intelligent Control Technique for Dynamic Optimization of...

Hi,

In short, this is due to the instability of NN learning (different initial weights lead to different local minima). Solution: run NN many times (e.g. 100 times) for each number of neurons. Then average results. Your graph will be nicer :)

Have fun

Well first the MSE increases as the number of neurons aren't enough, which leads to under-training .....then once the neurons are in the optimum range , the MSE starts decreasing..... finally,if you further increase, it leads much more number of neurons than required, which leads to over training..... I hope this helps you...