In recursive Bayesian filtering (Kalman, particle, etc) poor model parameter selection will lead to low quality tracking and eventually track loss. Is there any method for measuring tracking quality on the fly without using ground-truth. I know that measurements likelihood can be used to compare different models but I am looking for a general absolute measure.
You could look at the state covariance matrix diagonal. Are you trying to select a model based on observations? Estimate the process noise (for example, when motion is modeled via random walk)?
I have bad news to you. Conceptually, no absolute measure exists. Comparison of of different models you are mentioning is the best thing you can get.
It suffices to think about much simpler case of independent measurements of unknown length of a physical object. In frequency-based understanding of probabilities you will get correct answer in infinite run only (and life is finite) and moreover you have to neglect quantum effects.
In Bayesian setting you can only guarantee that you are gradually assigning higher probabilities (beliefs) to models, which are closest to reality but you have no guarantee that reality coincides with any model you consider...
One idea might be checking the normalized innovation sequence (or its sum for vector valued measurements). It is usually used in Kalman filtering applications for detecting changes in the estimated parameters. So it is also a good measure for understanding if the filter is losing the track.
Usually to get a good idea from these values you need to apply a low-pass filter. In case the filter loses the track the low pass filtered value, which is zero mean when the filter is working well, increases and somehow you can understand how much the filter lost the track by checking the magnitude of this increase. If the parameters for the filter are tuned correctly (e.g. a high process noise is selected to make the filter agile enough to track the changes in theparameters) basically you will not observe any change in the normalized innovation sequence values. Rather than appyling a low pass filter, using CUSUM might be also fit to your problem.
Please check
F. Gustafsson, Adaptive filtering and change detection. New York: Wiley, 2001.
and
M. Basseville and I.V. Nikiforov, Detection of abrupt changes: Theory and Application. New Jersey: Prentice Hall, 1993.
for more details.
My publication (specifically, Section III: Change Detection and KF Adaptation)
https://www.researchgate.net/publication/270454180_In-Orbit_Estimation_of_Time-Varying_Residual_Magnetic_Moment
may also give more clues about what I mean...
Article In-Orbit Estimation of Time-Varying Residual Magnetic Moment
I'm with Halil's answer here. You can check the residuals or innovations --- the difference between your measurements and their predictions --- for whiteness. If there's a correlation from time step k to time step k-1, there's a systematic error somewhere. From that test, you can at least detect filter divergence.
Thank you all for answers. I personally tend to believe that Miroslav is right that there cannot be an absolute measure. The Halil answer may only be applied to Kalman (Linear-Gaussian) case and it that case the normalized innovation is scaled log-likelihood of measurements.
However, Halil raised the idea that the sequence of innovations are zero mean when the filter works well. I guess this comes from the fact that he assumes the measurement noise is zero-mean Gaussian. It made me think that maybe we can relay on some properties of the measurement model. I think it can be generalized to other type of distributions by trying to understand how well the measurements are modeled by conditional measurement model given the filtered state. For that we need a measure that tells if a set of samples are from a given distribution.
- Badges
- Science topic