Hello. I think background subtraction is a simplified form of frame integration that assumes that the target object is not moving in the image. In general, if there is motion, you need to allow for possible target motions. The hidden Markov model framework is one such technique, although it is usually applied to point (single pixel) objects.To some extent, the detection and tracking problems are connected. Have a look for track-before-detect approaches, which look at multiple frames and declare a track present/absent after integration over several frames. In terms of processing requirements, there is a general property that more advanced algorithms (with typically better performance) require more processing. There is no way around this. The optimal approach (e.g. Bayesian) has unbounded complexity and so must be approximated.

Please have a look at the review article below for further suggestions and references. There is also a reference on the HMM technique itself, applied to a sonar problem.

Article Taxonomy of multiple target tracking methods

Conference Paper A High Performance 1-D Hidden Markov Model Tracker for Passi...

Could you (or perhaps somebody else) explain how the OpenCV background-subtraction algorithm works? I suspect that is the cause of your computational bottleneck. It can be a very expensive or very cheap (e.g. consecutive frame differencing) process, depending on how it is implemented. You need to make sure you are using a method that is appropriate for your particular background.  

Optical flow could be a useful input for both your background subtraction algorithm and your foreground object tracker.

I wouldn't try switching algorithms on and off, depending on your track state. Taking a 'systems perspective' I think it would be better if everything ran all the time  and design for the worst-case scenario.  

Small objects better be tracked with template matching or Bayesian segmentation based algorithms (depending on background and object properties)

P.S. This work could be interesting for you, this approach is more complicated when CV background extraction, but the extraction quality is much higher.

Article Extraction of the objects observed on a non-uniform backgrou...

I looked into the use of local sptatiotemporal filters for background subtraction in the 2nd attachment and fading memory polynomial filters in the 1st attachment. Both produce an optical flow field as a by-product. Both are reasonably efficient, and have the potential for realtime application.

Conference Paper Improved IIR Low-Pass Smoothers and Differentiators with Tunable Delay

Conference Paper Parallel software implementation of recursive multidimension...

Please see this video : https://www.youtube.com/watch?v=FpdGx6psnUA

This comes from our smart Logarithmic image sensor which can do the background subtraction directly in the pixel. If you are interested, you can buy this camera from us with USB interface and softwares. (www.new-imaging-technologies.com)

-yang

Hi Haziq, I tried the MOG in OpenCV some years ago, but I remember pretty terrible accuracy. Even, my background subtraction algorithm was superior compared to OpenCV MOG what was not a big magic. I assume you know some C++, I recommend you to check the bgslibrary: https://github.com/andrewssobral/bgslibrary . Andrew put an incredible amount of background subtraction algorithm in this library together and it is easy to evaluate them for accuracy/speed through the same API.

Hi Csaba,

I've been in this field for more than 30 years. Of course the background subtraction is a simple operation. But when you do this in pur software way, you need a complete digital processing system with DSP, FPGA, RAM, etc. This background subtraction is done inside the pixels of the sensor, so the only the image difference is sent to outside. No digital processing is needed and it consumes very few energy and also operates very fast.

Best regards