I have used traditional methods (guyan - serep - dynamic expansion - best achievable eigen vector ) but there is some error in final total mode shape which make divergence in damage detection.
I have never worked with damage detection but I have worked with other parts of your problem.
The thing with any energy based transform is that although it is100% perfect energywise, it always is wrong at a given time and place. The error can be small or large depending on how you make your transform. The use of modes is an energy based transform.
To start, you must first decide which best captures your problem the best - free-free modes or fixed modes? The reason why simply is that an incorrect choice of modal basis may lead to slow, if any, convergence.
For free-free modes, you can check convergence by summing the modal mass/inertia and comparing it with the physical mass/inertia . For fixed modes, you must compare with static stiffness. You will soon see that you never get a perfect match with any of the modal bases, in particular when looking at the phase for frequency response functions.
This leads to residue compensation. Again, you can never make the correction perfect, but you can make it close to perfect at a few positions, so you must decide where it does any good.
In essence, what you do is select the positions of interest, e.g. your sensor positions and/or interface positions in the case of substructuring.
At 0 Hz or very low frequency, you compute your response fields from unit loads (free-free modes) or from unity displacement (fixed modes) using a direct solution. You then compute the same thing using your modal basis. The difference between the direct solution and the modal response is your low frequency residue. Next you repeat the direct solution at a frequency above your highest frequency of interest and compute the high frequency residue.
If you have multiple load cases, you can run Singular Value Decomposition and extract linear combinations for the residues. This is called static modes in the Craig Bampton substructuring method. Craig-Bampton is the most widely used method in substructuring for simulation.
In any case, correct modal basis selction + residue compensation are the tricks of the trade to obtain accurate results.The area to look for best practice is for substructuring which can be model based, test based or model/test based.
In short, you are not alone. Many people have barked up this tree during the years.
Last, with reference to modes and damage detection, I believe you should take a look at the Chalmers reference and look at the concept of observability.