In pattern-recognition applications is common the use of log-sigmoidal neurons at the output of the NN to indicate a sort of "probability" of occurrence for events. Nevertheless, in a scenario in which the possible events must be ordered to select the most probable ones, linear activation functions could reproduce the same order (in fact, any monotonic function would maintain the same order). Therefore, the difference could be the simplicity since a linear neuron is easily implementable in a VLSI circuit, as an example, rather than a log-sigmoidal. Does this mean that log-sigmoidal can be replaced in any case? I think that probably no, since back-propagation training could be performed easily when a log-sigmoidal function is used so that training is performed with log-sigmoidal but for the final application, a linear one replaces this function.
The choice essentially depends on the problem to be solved, on the data coding method, on the method, the learning algorithm, on the neural network model itself, etc. In many cases, it is required to determine the winning neuron, so in such cases, in my opinion, it is more expedient to use nonlinear activation functions, and for simplicity of their hardware implementation, their simplified piecewise-linear approximations.
Modeling and possible implementation of self-learning equivalence-convolutional neural structures for auto-encoding-decoding and clusterization of images
- August 2017
- DOI:
- 10.1117/12.2276313
- Conference: Third International Conference on Applications of Optics and Photonics
The choice essentially depends on the type of the problem being solved, on the required performance and speed, on the accuracy of the calculations, on the data coding method, on the method, the learning algorithm, on the neural network model itself, etc. For many tasks, especially clustering, recognition, where determine the winning neuron, and ensure high speeds and very fast convergence of processes, the most appropriate, in my opinion, equivalence nonlinear activation functions, and for simplicity of their hardware implementation, their simplified piecewise-linear nye approximation. Try to read my publications.
- Badges
- Science topic
- Similar topics
- Organizational Psychology
- Training