SG filter, 2nd order derivative these all use for data smoothing. In pre-processing you have to use such techniques for reduce data dimensionality or data reduction. For that you can use principle component analysis (PCA).

Dear Somnath Paramanik

Thank you for your answer, but I think that you misunderstood my questions.

SG filter is a pre-processing tool as you said for smoothing and also for reducing baseline effect in the NIR spectra. PCA is an exploratory technique that can be use as you said for reducing the dimension of the multivariate data but also for "detecting outlier".

My questions are related to scikit-learn techniques for removing outliers from my data before processing with Machine Learning algorithms.

- Is enough to apply scikit-learn pre-processing technique (such as standardize, etc) to my data (including the NIR spectra) for removing outliers?

- Do I need to pre-process the NIR spectra separately and then standardize the whole data-set (including the pre-processed NIR Spectra)?

I hope that you understand my questions now,

Once more thank you for sharing your opinion

Yeah, before doing the processing, u do the normalization, standardization etc. You can do for the whole spectra or if the wavelength range is fixed for your application then go for only that part like only NIR.

I would like to update the new ideas I have been discussing with some expert in Machine Learning related to this discussion-question. I hope that they can be useful for all of the followers.

These are some of the suggestions I have received:

- To carry out the Machine Learning (ML) algorithms using the raw NIR spectra

- To carry out the ML algorithms using the pre-processed NIR spectra:

- By the 2sd derivative of SGolay

- By the Wavelet-transform method

Wish you all the best

Dear Edenio,

In any robust optimization it is important to minimize the dimensionality of the optimization {i.e. reduce the number of parameters to be optimized). Design your experiments if you will by accessing the unique effect of each parameter and holding constant as many of them constant as possible. In terms of experimental design, you treat the spectra, colour and physical properties as component causes and adjust them appropriately.