I want to discuss about a design that I've never seen before anywhere, and that is to define a node as a PWM generator in a neural network. First things first, this design is a really hungry resources design, I know. But just as a concept (or even more than that), what if we replace the current's amplitude with the duty cycle? Here, in a fully connected network design (probably) we have to define some fuses (instead of having weights) that totally intercept the communication between two nodes and to consider each node's input as an OR gate. Now, the output of each OR gate should be connected to the node's main PWM generator's enable (start) or clock terminals (using clock terminal may help us to reduce the complexity of the overall design, but not necessarily has a great impact). I think we can consider this design something in between spiking and artificial neural networks (as in the case of spiking we can define a threshold like as the duty cycle or whatever else, but the point is it's not an always-on design so if we become very optimistic, it can reduce the energy consumption somehow).
What I'm seeking now is that, is there anyone who has an opinion in agreement with this idea? Does it have any worth to be implemented?
Hi Alireza Alikhani ,
just some raw thoughts, inspired by your idea:
"Energy consumption" sounds like hardware implementation, not just simulation. PMW (or frequency) has the advantage of avoiding analog signals, and the measuring of the duty cycles could be a simple counting process. So, before programming an FPGA or something similar, one could just take some hundred really simple and cheap microcontrollers, one for each node. Then, the usual problem arises: We would need many inputs per node, but small and cheap microcontrollers just have 6 or perhaps 12 I/O pins. This leads to serial busses etc.
But many natural neurons are just collecting the input signals by one long axon each, seemingly without any formalities. Transferred to PWM signals, and limited for the moment to layered networks, each node which gets its input signal from the nodes in the preceding layer could do so via a single input, with all open collector (or open drain) outputs of the preceding nodes wired together (that's your OR or AND gate, but only one per layer). In order to avoid to much masking, each node's output should deliver only a small duty cycle.
Now just for summing up, all nodes could work asynchronously, and the result would be statistically spreading. But to introduce weights, each transmitting node could "own" a time slot, and each receiving node "looks through bars", i. e. masks of about the same duration as the maximum pulses can be shifted in time during learning.
For example, there are some layers of 50 nodes each. Each node is clocked by an internal 8 MHz clock, PWM frequency is 1 kHz, so each time slot is 20 mys long = 160 clock periods (so the resolution is better than 7 bit). Between each layer, there is one wire connected to 50 outputs and 50 inputs, and a second wire, signaling the start of the time slot of the first transmitting node to the 50 receiving nodes. The transmitting node contributes a pulse of a certain length, and after 160 clock periods triggers the second transmitting node. Regarding the transmission, only 1 of 50 nodes had to be awake; but the receiving process is a full time job. One could try to declare the masked time intervals as sleeping time but since there are many (in this example, I would start at least with 25) I guess it would not matter much. All the receiving nodes differ just by the temporal positions of their masks.
Advantages: Pure digital and quite simple nodes, no precision central clock necessary, just two wires between adjacent layers (or, without layers, just two wires (= 1 axon) per node).
As you see, your approach offers enough material to think about for a while. I leave the question of how to train such networks to you. ;-)
Thanks Joerg Fricke for some raw calculations. I look at it as a concept that may have some fun inside it. I don't think training it be such a big deal (indeed it is but a the end of the day, i don't feel it to be very different from the other types of NNs). Anyway, just multiplying your calculations to 1000 (which is totally okay considering simple MCUs like dsPIC) we can gain some plausible resolutions.
At the end, thanks for sharing your thoughts with me, I'm glad that I found someone else behind this idea!
Dear. Interesting idea. Deserve to be tested. You may get a good result. Effectively, in spike coding, neurons respense has some kind of frequency modulation. PWM is an exemple of frequncy modulation based signal amplitude. Try it for exemple on object recognition. Waiting for a research paper, before others do it!
- Badges
- Science topic