When I try to analyze the navigation for Mars approaching phase (entry 24h to entry), I find a significant divergence of rv in the last half hour. I used the X-ray pulsar measurements as observations, which could be simply modeled by the position vector projected in three different direction vectors. I have tried two kinds of filters: EKF and EnSRF(Ensemble Square-Root Filter), but no obvious difference was found. However, if the filtering step is shortened, the divergence will be less significant. On the other hand, if I use the orbit elements as states, all six orbit elements converge well. However, the rv derived from orbit elements still diverge. Up to now, I believe that the strongly nonlinear dynamic system may be the main contributor to this situation. I wonder if there are any advanced filter algorithms dealing with highly nonlinear systems with limited filtering steps?
My conference paper also pointed out this problem, but it still has not be solved.