In the first paper I linked it states that the activation function for the XOR problem is a step function, and in the second paper I linked the activation function is a logistic function. So, I was wondering, the activation function is problem specific? I am asking because I am trying to figure out why my implementation fails to learn the solution to XOR. In the second paper i linked, it states "the search range is set from -2 to +2", I haven't figured out - the search range of what - the weights?
Thank you
http://www.complex-systems.com/pdf/02-6-2.pdf
http://www.sciencedirect.com/science/article/pii/S1877705812019686
No, the activation function isn't problem-specific: it is possible to use either step functions or logistic functions for the network-it's the architecture of the network that affects its ability to represent a rule. Of course the tuning of weights necessary to realize a given rule depends on the activation function. What is sought is a point in the space of weights-so the interval refers to them, of course.
I thought as much, however, I don't know why my implementation can't find the weights necessary to solve the XOR problem...
My implementation is as follows:
Initialize weights to continues random values in the interval: [ -0.1, 0.1]
Initialize T to 10
Set cooling factor to 0.85
Calculate error (on entire dataset) for the set weights - call it E_old
repeat K times {
update all of the weights by some continuos random value [-0.4, 0.4]
calculate the error (on entire dataset) with the new weights - call it E_new
set delta_error to E_new - E_old
if delta_error cooling-factor * T
Please, ignore the numbers, I am curious if my implementation is ok.... it didn't occur to me before, but the problem may be because I am calculating the error on the entire dataset, not just the shown examples... ?!
It should work-only the cooling schedule needs tuning and the interval in which the variation of the weights is selected-just like in any Monte Carlo simulation. However I don't understand what's the difference between ``entire dataset'' and ``shown examples''. If you want to learn the XOR, you find weights based on a dataset. That's learning. If you then want to check whether the network, having been ``trained''-i.e. having learned the examples-can represent others that's something else, generalization-a distinct problem.
Both assume, however, that the network can represent the association at all: so the XOR rule requires a two layer network: a perceptron simply can't represent the XOR rule at all.
Cf. here: https://hal.archives-ouvertes.fr/jpa-00211064/document for a worked example of simulated annealing and tabu search for the binary perceptron.
The error function is a measure of performance of the trained weights, I was thinking if it should take into account the examples presented to the network or the entire example dataset (which includes the exampels not presented)...
I know, I have a network with two inputs, one hidden layer and two bias nodes - one connected to the hidden layer nodes, and the other connected to the output nodes...
EDIT: Found out I was calculating the error the wrong way.
- Badges
- Science topic