Imagine a QPSK constellation. If say there are a few bits in error (close to the center) but the spreading of the constellation (around +/- 1 +/- j) is tightly bound. This can lead to a high BER but low EVM.
EVM just gives you the average difference of received points compared to the ideal point of a constellation diagram (E.g., provides information about first order spreading). If your distribution of points can be described closely by a Gaussian distribution, EVM could be calculated from mean and std quite accurately, and would have a close relation to the actual BER.
If, however, you have a case where only a few points are far away from the ideal point, while all the others are close to it (as you described in your case), EVM does not relate well to BER. Similar discrepancies could occur for other statistical distributions such as truncated Gaussian, or bean-shaped due to NL-phase noise.
So I think, BER is the only true performance measure - EVM could serve as estimate for certain cases, but would fail for others.