It is a huge topic, as there exist some open libraries for implementing the modular exponentiation on FPGA, there must be some experience in optimizing such libraries.

In the software implementation context, I've some experience, I've achieved up to 90% of time improvement in applying Montgomry Multiplication and the Chinese Remainder Theorem for the computation of the modular exponentiation, comparing with traditional methods.