Greeting,
I am doing data analysis on NIR of apple. At this time, I know how to use Unscrambler and have knowledge of the process and how to deal with the problem to improve the results by distinguishing outliers and so on.
Here is my question: "How can I use preprocessing to improve the results?" I assume I have some knowledge when to use various filters. However, some cases when I apply SG Derivative, MSC or SNV filters, my results are not very well improved or sometimes get worse. I need some guide or clue how to use preprocessing of data in my PLSR analysis of NIR data.
“When I use 2nd order derivative SG filter (transform) the values of intensity will be totally changed (as in Figure 2). Therefore, after applying the PLSR on this filtered data, the results will be completely different in regression coefficient values (in Excel file regression coefficient for NIR raw data and filtered data are shown in sheets 1 and 2 respectively)."
Would you please show me with some examples how to use this kind of filtering to improve the results?
Thank you for your attention.
Dear Mohammadmahdi Davoudi
Your pre-processing result is normal. After processed your data with SGolay 2sd order, every positive peak in the raw spectra will goes to the negative in the transformed spectra (Pre-processed with SGolay 2sd Order).
Related to the Regression results after PLS regression, it is normal that the PLS result using the transformed spectra differs from the PLS result using the raw spectra. But, you must check if the regression has improved or not using the transformed spectra (Looking at your spectra plots. It should improve).
What you can do? Please, carry out PLS using the raw spectra. Then carry out PLS but now using the Transformed spectra with SGolay 2sd order (I recommend you to insert minimum of 7 points in the unscrambler (I use Unscrambler) for right and left). Also, you should vary the left and right point of the derivative and compare the PLS results among them.
I am not expert on apple, but it is recommended to use only a pre-processing filter. However, because of the noise on the spectra (like yours), some authors recommended to carry out two filter techniques such as SGolay and SNV or MSC.
Wishes you all the best. If you have some doubts dont hesitate to contact me.
- Badges
- Science topic
- Similar topics
- Mathematics
- Statistics
- Regression