I am trying to model a neural network where the output of the network is checked and assigned a score based on the gold data. For example: if the predicted label of the network is in a predefined set, then the loss should be low (for example loss = 0.0), otherwise, the loss should be high (for example loss = 1.0).
I wonder how I can model this loss function in order to apply gradient descent to update network weights. As my understanding, we usually calculate loss directly from the predicted output, which is different to my case because I calculate the loss with a condition on the predicted label.
The workflow of my network is like this:
predicted_label = NeuralNetwork(input)
if predicted_label in [A, B, C]:
loss = 0
else:
loss = 1
Have you ever seen similar models before? Or do you have any suggestion for modelling this kind of network?
Thank you very much in advance.
If you want to do a simple classification with your ANN (and if we assume you have a quite limited set of categories - as in your example) then you can use the "standard" SVM and Softmax data loss functions. That is in case with "loss" you refer to information and data loss that occurs when comparing the output/prediction with the actual data.
In your pseudocode I can see you are assigning a loss of 0 or 1, but its not quite clear if you accumulate this for every case in the training set or what exactly do you intend to do afterwards with it?
Actually I want to update the parameters in my network based on this loss. Because this loss is non-differentiable, I can not use gradient descent to do that (right?).
I have found the keyword "non-differentiable cost function" and did some studies on it. Hopefully I can find something useful for my research.
If you still have a limited number of points where the loss function is non-differentiable you can add "approximation" for the loss function in these points to make the function differentiable (check for example simple kinks and sub-gradients).
- Badges
- Science topic