I have a 3 layer MLP architecture with which I want to approximate models for certain logics. I know that these models can be approximated using only positive weights and this property of the network would be very advantageous for rule extraction.
Unfortunately, when limited to positive weights, the standard Backpropagation algorithm tends to get stuck trying to reach local minima with partially negative weights.
So I am wondering whether there are any learning algorithms or variants for this special case which I have missed in my research so far.
Any advice would be greaty appreciated.
.
why not encode your weights as w = exp(v) and apply the backpropagation algorithm up to v ?
(of course, any function from R to R+ would do the job too)
.
In Rosenblatt's original Perceptron paper, he talks about training MLP's with only positive weights, so maybe you can find something interesting in there?
The paper in question is: The Perceptron: a Probabilistic Model for Information Storage and Organization in the Brain, by Frank Rosenblatt.
.
why not encode your weights as w = exp(v) and apply the backpropagation algorithm up to v ?
(of course, any function from R to R+ would do the job too)
.
In this paper, they use a barrier term within their cost function, this could probably be applied to your cost function, too:
http://jmlr.org/proceedings/papers/v29/Nguyen13.pdf
It springs to mind neural network. Perhaps CGP could be helpful too. It can produced digraphs. It has been used to evolved circuit, perhaps it can evolve too MLP too.
However, are you trying to evolve the algorithm or the solution of the problem?
Patricia
In relation to the last couple of answers, you might want to take a look at the more general field of "neuroevolution" which uses various forms of evolutionary algorithm to train neural networks. NEAT is particularly popular, and has software support. However, there are various other algorithms, including the variant of CGP that Patricia mentioned. Basically you would have to modify the initialisation and mutation routines so that they only generate positive weights.
Thank you all for the many inspiring suggestions!
After posting this question another part of my project went sideways and I've only now gotten the chance to tackle the problem. Because it seemed like the most simple solution, I took Fabrices advice and mapped the non-bias weights to x^2 (I'll try exp(x) for comparison later), adapting backpropagation accordingly, which was surprisingly simple. The results seem to be quite good so far. So in case someone with the same problem finds this thread later on, I can only recommend Fabrices approach.
Hi Fredrikk
I am also working on an ANN with 2 iputs and 2 outputs, 1 hidden layer. I am also applying backpropagation to train my network and just like you, I end up with negative weights and bias for some neurons.
I am also trying to implement the method you tried that seemingly worked (Fabrices advice) to obtain positive parameters.
I know its been a while since you posted this, but could you elaborate more on how you mapped the weights? Thanks in advance.
Hi Nima,
My BSc thesis (which this was for) is available on GitHub, along with the code. You can find the derivations for backprop with unipolar weights in the appendix.
https://github.com/frhrdr/Lukasiewicz-Cores/
Basically I replace each occurrence of weights w with another squared variable w=v^2 (alternatively w = exp(v) or w=softplus(v) are also options) and adapt the derivatives in the backpropagation updates and activation functions accordingly. This way, the stored v can still be negative (which might be the source of your confusion) but the computed w=v^2 cannot.
And just to clarify, restricting ANNs to positive weights restricts their expressiveness and is not commonly done, unless you have a good reason for doing so.
Let me know if you have any more questions.
Hi Fredrik,
Thanks a lot for reaching. your help is really appreciated. i noticed in the appendix of the paper, you provided net input to each neuron as following:
Neuron net input = x0 * w0 + sum(Xjk * 1/2(Wjk)^2 )
Is Weight = 1/2(Wjk)^2 ?? in other words, did you mean from your post that V = Wjk?
Hi Fredrik,
So i have implemented the V^2 variable into the expressions containing "W", except the weight update.
At the mean time, I have also changed the weight update equation, hidden layer delta error, and output layer delta error accordingly, as the paper specified in the Appendix. I was wondering if the bias update equation is supposed to be similar to its original case?
From the results, I obtained positive values for the weights and bias as anticipated. However, My output results are a bit weird, and I feel my trained network fails to generate the optimal results (or at least sensible output).
Is there something that I might be missing? Because I am not really sure how to troubleshoot this problem.
I have included my code as well. I am using Processing software for my neural network construction and training. The code is a bit long, but i have also provided a link which described various classes in the code. Warm Regards
http://arduinobasics.blogspot.ca/2011/08/neural-network-part-7-cut-and-paste.html
- Badges
- Science topic