I suggest you follow as almost similar query asked here : https://www.researchgate.net/post/Which_activation_function_should_be_used_in_a_prediction_model

Moreover., you may follow https://medium.com/the-theory-of-everything/understanding-activation-functions-in-neural-networks-9491262884e0

https://www.geeksforgeeks.org/activation-functions-neural-networks/

Simply add the tensors of the previous output and the current input and perform a sigmoid or tanh activation on that vector. Basically the "gate" which you will be designing will take two inputs: (i) current input and (ii) previous output. The gate will perform two operations and in this order: (i) addition of the two vectors, that is current input+previous output and then (ii) sigmoid/tanh function applied to the vector obtained from previous step.

Muhammad Ali and Rajdeep Kumar Nath Thanks for the suggestions.

I wanna ask you Rajdeep Kumar Nath how to implement the thing with python? I mean, can it be done with Keras? or Pytorch will be better choice.