Well, first of all, I know this is a pretty basic question, but here’s my problem: I’m working with a vis-spectrum (400–700 nm), and when I try different noise reduction techniques such as Savitzky–Golay filtering, moving average, and PCA-based denoising, I still see wiggles in my spectrum (as shown in the attached pictures).
So I would really appreciate it if someone could suggest any method to eliminate these wiggles. Also, I’m doing this in MATLAB using sgolayfilt(2,15), movmean, and a small script I wrote myself for PCA-based data reconstruction using two principal components.
An S-Golay filter implements the principle of convolution and is a type of finite impulse response (FIR) smoothing filter. These filters commonly produce such 'artifacts' or comb structures due to the zeros in the transfer function. Without going into too much detail, try to change the order of the convoluted polynomial or increase the window length. Look at the animation on Wikipedia for a better understanding of the order and the length:
https://en.wikipedia.org/wiki/Savitzky%E2%80%93Golay_filter
Choosing the polynomial order is dependent on the shape of your original data.
Look here to understand polynomial order:
https://pisszentb2lcircuit.z21.web.core.windows.net/sign-diagrams-polynomial-functions.html
If the length of your window is too small a high order polynomial will be fit into a smaller space. The Sgolay filter takes the peak of this. Convolution shifts the fit curve (the polynomial) and therefore the frequency of the polynomial pops out in your result. The frequency is related to the period or your window length. I hope this makes sense?
Always compare your resulting filtered data with the original. Filtering always creates a lagging result (the peak may be shifted to the right) - its good to indicate how much your filtered result deviates from the orinal and the possible phase error, especially when searching for a peak in a visual spectrum (e.g. +/- 2 nm on the horizontal axis and showing an error or band of uncertainty around your smoothed data, here it makes sense to analyse many plots to analyse the statistical behaviour - this is also inherently a type of averaging which is synonomous to filtering)
Petrus Pieterse
Thank you for your really detailed answer. After revisiting my code, I realized I was applying the filter on the wrong dimension of the data. Once I fixed that and adjusted the polynomial order and window size based on your suggestions, the comb effect was eliminated.
- Badges
- Science topic
- Similar topics
- Computer Communications (Networks)
- Simulators