For example when you want to design a high speed response controller with minimum steady state error ( in network modem) the choose of a high-gain is a good thing, but what are the caveats from the viewpoint in control theory?
So far I understand you need to match the gain of a system to the input of another system that needs to be controlled. You can have very high gain but that rises the expenses of the amplifier. Also you can match impedance between the output and input of the two systems, which produces maximum power as a rule of maximum power transfer theory.
The process of setting the optimal gains for P, I and D to get an ideal response from a control system is called tuning. There are different methods of tuning of which the “guess and check” method and the Ziegler Nichols method. I think The gains of a PID controller can be obtained by trial and error method...
gain in PID usually has steady-state errors (the difference in set point and actual outcome) unless the control gain is large. As the control gain becomes larger, issues arise with the stability of the feedback loop. For instance, reducing the rise time implies a high proportional gain, and reducing overshoot and oscillations implies a small proportional gain. This is not possible to achieve in all systems.
There are different methods of tuning "trail an erro" method, the Ziegler Nichols method etc. You can use Ziegler Nichols method to find what will be the maximum gain.