I am developing PLS-R calibrations based on NIR spectra for estimating wood density. When I used a totally independent set of 18 unknown samples to validate this model, I found a negative R² for describing the relationship between lab-measured and NIR-predicted data. What does it means?
Please find enclosed the results.
What about RMSEP and representative subset selection?
Did you try for instance the Duplex algorithm to design test set? You develop calibration model in a relatively wide range, but your test set is not spanned over calibration domain. It is not a remedy but at least you will get more info about model's performance.
Another problem can be due to a presence of in a calibration set some samples that have an impact on the construction of the calibration model. Then, you could think of construction of robust model (e.g. robust version of PLS called partial robust m-regression).
A possible problem can be related with representativeness of your calibration samples (and it seems to be the most probable one). You are focused on natural samples (wood) and therefore you can expect relatively large differences in properties from batch to batch. Apparently, new samples contain new sources of variability that are not considered at the calibration stage.
How did you select complexity of your model?
Look also at papers by Paul Geladi - he was also busy with calibration of wood samples and hyperspectral imaging.
Please have a look at the following references:
Article TOMCAT: a MATLAB toolbox for multivariate calibration techniques
Article Representative subset selection
Article Controlling sugar quality on the basis of fluorescence finge...
Dear Paulo,
irrespective of what Unscrambler says, per definitionem R² can not be negative and also the absolute value of R² can not be > 1 as R² is explained variance divided by total variance. So first of all you should copy reference and predicted values to another software (e.g. Excel) and recalculate all statistical parameters of the regression (RMSE, R², slope, offset). Obvious something is wrong with the calculations done by Unscrambler.
Furthermore, your validation samples seem to cover only a small range of the calibration data which reduces the information value of your validation.
Kind regards
Dieter
Dear Guillaume,
due to Unscrambler's method references R square for calibration data is calculated as percentage of explained variance and has to be always between 0 and 1:
R-square
The R-square of a regression model is a measure of the quality of the model. Also known as coefficient of determination, it is computed as 1 - (Residual Y-variance), or (Explained Y-variance)/100. For Calibration results, this is also the square of the correlation coefficient between predicted and measured values, and the R-square value is always between 0 and 1: the closer to 1, the better.
The R-square is displayed among the plot statistics of a Predicted vs. Reference plot. When based on the calibration samples, it tells about the quality of the fit. When computed from the validation samples (similar to the “adjusted R-square” found in the literature) it tells about the predictive ability of the model.
However, I already had some strange results in validation statistics of Unscrambler. Recalculating the results give more plausible ones. But thanks for the interesting explanation of Stone-Geisser criteria.
Kind regards
Dieter
Dears,
Thank you for your suggestions. That model was built using 17 wood species and the 18th "unknown" species was used in test set in order to verify the robustness of the model in predicting the wood density of an "unknown" species. I repeated this approach 18 times. In short, some "unknown" species can have their density values predicted by PLS-models with good agreeement (high R² and low RMSEP). However, in some cases (like as attached print screen), poor predictions are achieved.
The hypothesis of Michal Daszykowski is correct: "Apparently, new samples contain new sources of variability that are not considered at the calibration stage".
I will "copy reference and predicted values to another software (e.g. Excel) and recalculate all statistical parameters of the regression (RMSE, R², slope, offset)." as suggested by Dieter Lohr.
Guillaume Hans, I've never heard about "Stone-Geisser criteria" before; Thanks for these information.
- Badges
- Science topic
- Similar topics
- Filing