Dear experts in Machine Learning,
As you all know, pre-processing the data set is an important step for obtaining robust results in any machine learning algorithm. I know that scikit-learn can be used.
I usually pre-process the NIR spectra with 2sd derivative of SGolay and then I carry bout PLS (Partial Least Squares regression). After, while developing the regression, I removed outlier samples based on the large leverage and high residual variance. However, because I am a beginner in Machine Learning I wonder how I can pre-process my data, in particular, the NIR spectra:
1- Do I have to preprocess the NIR spectra separately?
2- Is Ok to process the whole set of data NIR spectra, color and physical properties with the Standardize of scikit learn?
3- Do you have any experience pre-processing such type of data set (NIR spectra, color, and physical properties)?
Any example, book, paper, link, ect. will be appreciate.
Thanks in advance
SG filter, 2nd order derivative these all use for data smoothing. In pre-processing you have to use such techniques for reduce data dimensionality or data reduction. For that you can use principle component analysis (PCA).
Dear Somnath Paramanik
Thank you for your answer, but I think that you misunderstood my questions.
SG filter is a pre-processing tool as you said for smoothing and also for reducing baseline effect in the NIR spectra. PCA is an exploratory technique that can be use as you said for reducing the dimension of the multivariate data but also for "detecting outlier".
My questions are related to scikit-learn techniques for removing outliers from my data before processing with Machine Learning algorithms.
- Is enough to apply scikit-learn pre-processing technique (such as standardize, etc) to my data (including the NIR spectra) for removing outliers?
- Do I need to pre-process the NIR spectra separately and then standardize the whole data-set (including the pre-processed NIR Spectra)?
I hope that you understand my questions now,
Once more thank you for sharing your opinion
Yeah, before doing the processing, u do the normalization, standardization etc. You can do for the whole spectra or if the wavelength range is fixed for your application then go for only that part like only NIR.
I would like to update the new ideas I have been discussing with some expert in Machine Learning related to this discussion-question. I hope that they can be useful for all of the followers.
These are some of the suggestions I have received:
- To carry out the Machine Learning (ML) algorithms using the raw NIR spectra
- To carry out the ML algorithms using the pre-processed NIR spectra:
- By the 2sd derivative of SGolay
- By the Wavelet-transform method
Wish you all the best
Dear Edenio,
In any robust optimization it is important to minimize the dimensionality of the optimization {i.e. reduce the number of parameters to be optimized). Design your experiments if you will by accessing the unique effect of each parameter and holding constant as many of them constant as possible. In terms of experimental design, you treat the spectra, colour and physical properties as component causes and adjust them appropriately.
Cite
Can you contribute to the discussion?
Reply
- Badges
- Science topic