In addition to what have been said above, using grout can change the structural behaviour of the prestress forces. It can change the forces form two forces at the beam ends to uniformly distributed forces along the whole length of the beam. This can only occur if the ends constrained are released!
All arguments mentined are correct. However, the main task of grouting the pipes with mortar is to produce a composite effect. As a result, later local damage or failure cases have no effect on the overall function of prestressing.