As explained by Prof. Harkouss, the improved MUSIC algorithm he shared includes forward-backward smoothing (which is a trick by providing double spatial subarrays to smooth over). The backward smoothing is done as he wrote:
R=R+J*conj(R)*J; where J is the backward identity
This aformentioned step is referred to as backward smoothing.
For example in a Uniform linear Antenna (ULA) Array setting, the good thing is that the difference between any two overlapping subarrays is a phase difference. Thus smoothing could be applied.
In a circular antenna array, it is NOT. But, there was some work done on circular antenna arrays (1990s) in which they could, indeed, be pre-processed to form a virtual ULA, where afterwards the smoothing remedy could be applied.
For coherent signals (from a source due to multipath fading) or correlated sources you should do a pre-processing. It means you have to change correlated signals to uncorrelated. To do this, one solution is spatial smoothing. Forward-Backward spatial smoothing is the best one. For more details you can see following references:
Shahriar Shirvani Moghaddam, Ali Janan, “Performance Evaluation of 2-D DOA Estimation Algorithms in Noisy Channels,” International Journal of Sensors, Wireless Communications and Control (IJSWCC),” Vol. 3, No. 2, pp. 95-100, Dec.2013.
Shahriar Shirvani Moghaddam, Somaye Jalaei, “Determining the Number of Coherent Sources Using FBSS-based Methods,” Frontiers in Science, Vol. 2, No. 6, pp. 203-208, December 2012.
Depends on what methods you want to use and what arrays you use. For example, if you want to use subspace methods, then you need to exploit some shift-invariance properties in the arrays. Then you need to look at the arrays that you use. If there is naturally a shift-invariance strcture in the array, like ULA and EM vector-sensor array, it can be easily be done via smoothing. If there is none such array, you may switch to the temporal structure or high-order statistics of the sources to look for this shift-invariance structure. If you use large scaled array, maybe you don't need to do anything on the source dimension, instead, you may want to do hankelization on the space mode and use tensors. I think there are plenty of methods to deal with coherence, and this is totally dependent on the specific applications.
In many applications, we need high resolution estimation of DOAs.
For this purpose, many high resolution subspace-based algorithms
such as MUSIC and ESPRIT have been proposed in the past. A new alternative effective method for DOA estimation for coherent sources has proposed ,which is also computationally efficient as a ( Bartlett - MUSIC ) method . While the algorithm has no limitation on the number of coherent sources , it does not need extra elements either.
Hence it is expected that this algorithm performs better in multi-path situations
when compared to MUSIC and ESPRIT.
This method takes advantage of Bartlett and Beamspace MUSIC methods and offers a low computational algorithm for locating the coherent sources.
In this method , Bartlett is used as initial DOA estimator for beamspace MUSIC method. Beamformer section forms a small number of beams in the adjacent region
of the Bartlett spectrum peaks. Then the outputs of beamformer are applied to MUSIC
algorithm and finally DOAs are determined with high resolution.