I am not sure if I understand your question completely. I suppose your question is in general about averaging technique used for calculating FRFs so that they have free of random noise errors.
It is correct that averaging is done to reduce the noise such that the averaged FRFs are smoother and more importantly free of random noise. However, there can be other kind of errors, which are not taken care of by the averaging procedure and require other signal processing techniques.
Please refer any standard modal analysis text for more details on averaging and other signal processing techniques used for reducing the effect of noise on the FRFs.