I am working on an artificial neural network model based on multilayer perceptron. For my model I am using Encog artificial neural network programming library. I am searching for way to set up different activation function for each neuron in my multilayer perceptron. At the moment I have found only way to set up single activation function for each layer, but not for each specific neuron.
Do someone has experience with such problem? If you have any, please share.
In case you use Encog in Java, refer to:
- either Section "4.1 Constructing a Neural Network" and following ones from J. Heaton, "Programming Neural Networks with Encog3 in Java", 2011 -
Book Programming Neural Networks with Encog3 in Java
- or Chapter 3 from J. Heaton, " Programming Neural Networks with Encog 2 in Java ", 2010 - https://universalflowuniversity.com/Books/Computer%20Programming/Neural%20Networks%20and%20Deep%20Learning/Programming%20Neural%20Networks%20with%20Encog%202%20in%20Java.pdf
- or Appendix B from J. Heaton, " Automated Feature Engineering for Deep Neural Networks with Genetic Programming ", PhD Dissertation, 2017 - https://www.heatonresearch.com/dload/phd/jheaton_dissertation_10259604.pdf
A priori, it is not possible to customize the activation function on a per neuron basis...
Thanks @Stéphane Breton. I was looking directly in the Java classes and I have found that it is not possible. My research is related to ideas where each neuron can have different activation function.
Please, suggest better software tool?
Sorry, I am not able to suggest a library that meets your specific need. As far as I'm concerned, I develop my own computational codes...
Neverthless, you may have a look at the leader libraries. Follow https://dzone.com/articles/progressive-tools10-best-frameworks-and-libraries.
- Scikit-Learn seems to have the same limitation as Encog. Check out https://scikit-learn.org/stable/_downloads/scikit-learn-docs.pdf
- Same with respect to TensorFlow... Check out https://www.tensorflow.org/tutorials/eager/custom_layers
An alternative is to implement your own solution by taking inspiration from such websites:
- https://cognitivedemons.wordpress.com/2017/07/06/a-neural-network-in-10-lines-of-c-code/
- https://machinelearningmastery.com/implement-backpropagation-algorithm-scratch-python/
I am not familiar with Encog. But if you are looking for other neural network frameworks, some of them provide concatenation layers so you can merge two or more group of neurons with different activation functions in a layer. For example in CAFFE you can use "Concat layer":
http://caffe.berkeleyvision.org/tutorial/layers/concat.html
- Badges
- Science topic