In general the RMSE is the main parameter to be compared.

According to my experiences various evaluations of the result are not necessarily consistent. For optimization, either a uniform measure should be (pre)defined (which still can be different for various prediceted parameters) to efficiently compare the various possible results, or the expectation concerning the predicted features of the target variable should be (pre)defined (e.g.: its extreme values should be kept/minimal smoothing is accepted/smooting is expected).

Hi Moshood

You cannot evaluate the goodness of fit of a variogram model from the cross-validation of the interpolation (kriging) as they are mutually exclusive. The measures from krige.cv tells you how good your interpolation model is (universal or ordinary kriging) not how good your variogram model is! In order to check how good your variogram model fits to the experimental variogram (which is the goodness of variogram model) you may use SSErr as it is the widely used and reliable measure. It's easy to calculate:

attr(output of fit.variogram, "SSErr")

Alternatively, you may try the maximum likelihood approach (function "likefit"). Please remember that a better variogram might not give you a better interpolation result as it is influences by many other properties e.g. data characteristics.

Hope that helps.

These diagnostic measurements are basically different. While SSerr is an indicator of the "goodness of fit" of the variogram model adjusted on the experimental variogram, ME and MSDR are applied after estimation to evaluate the prediction accuracy. SSerr could help you in choosing a priori the optimal variogram model (theoretical) which best fits your data, but after performing the prediction and the cross validation you can obtain lower ME, RMSE and MSDR values because the utilised model is not the most performing one for prediction with your data in your area. Thus, you can conclude that spherical model is the optimal one in your case.

Cross-validation is a method to evaluate the "predictive" abilities of your model. Where "model" refers to the entire geostatistical model (i.e. the mean and dependence structure) and not just the variogram model. As Cressie (1993, p.101-104) points out, cross-validation is used to assure your model predictions are okay, it is not necessary meant as a tool to select variogram models (spherical vs exponential). To compare models by cross-validation, you may use the mean squared "prediction" error (MSPE).

To choose between variogram model structures (e.g. spherical vs. exponential) you might want to employ the AIC or BIC model selection criteria.

For each point, divide the residual obtained by (leave-one-out) cross validation by the standard deviation of the kriging prediction error. The square root of the mean of the standardized residuals squared should be close to 1. If less than 1, the standard deviation of the kriging prediction error are too low and vice versa.

It looks rather suspicious to me that the best-fitting semivariogram model (exponential) does not return the best MSDR values (except on year 4), and also the best cross-validation results.

Are you sure that you used the same number of lag intervals (i.e the same number of variogram points) to evaluate all the semivariograms and SSerr? It is probably better to use also here an "averaged" version of accuracy indicator, like RMSE or MAE, dividing by no. of lag intervals.

In any case, we are typically interested in prediction accuracy, not in semivariogram modeling accuracy! So I would say that on average the spherical model wins over the exponential model (except for year 1, where Gaussian model gives the lowest RMSE).

Both ME (or MBE, mean bias error) RMSE are measures of prediction results accuracy. In contrast, MSDR states how well the kriging-predicted variance reproduces the variance of the observations (MSDR tending to 1). In fact, this is a measure of the goodness of the kriging model rather than of the prediction accuracy itself.

I agree with that ,ME, RMSE, MSPE, etc are very useful, but I also use the R-squared of the regression between observed and predicted values and its scatterplot, they are very intuitive.

Cross-validation gives a measure of the "global" quality of the estimates obtained by, for instance, ordinary kriging. Kriging results depends from variogram model (so cross-validation results depends on variogram model) but also depends from the geometry of samples used by each kriging estimate. And depends also on the kriging method chosen. RMSE is a good syntesis of the this overall quality of the predicton process. Scatterplot of observed versus predicted values is very useful. However R-squared can´t evalutes biased results.

In the link below, an article that uses cross-validation and variogram model testing. It may help.

https://www.researchgate.net/publication/258225101_Geostatistical_analysis_using_GIS_for_mapping_groundwater_quality_case_study_in_the_recharge_area_of_Wadi_Usfan_western_Saudi_Arabia

Article Geostatistical analysis using GIS for mapping groundwater qu...

I agree the both metrics are useful, though in my experience cross-validation is the more robust measure.

One should be very careful with using statistics similar to R^2 for comparing the experimental variogram vs. model variogram. The R^2 measure assumes that the "data" (i.e., the lag / variance points) are iid gaussian (they are not - actually, the "raw" variogram should have a distribution more like a chi-squared). Also, regression on the experimental variogram can mask the fact that you have fewer points at larger lags - i.e., you should not try to fit the large-lag parts of your variogram as closely as the short-lag parts.

There are many heuristic metrics out there for how to alter the R^2 metric a bit so that it makes more sense for fitting experimental variograms. However, a generalized cross-validation is much more theoretically consistent.

It is an illusion to think that any of the various statistics will identify/determine the "optimal" variogram model. At best they allow the user to compare several models but the various statistics vary in how sensitive they are to various changes. For example the experimental/empirical variogram is not unique for a given data set, the user must choose the width of the distance classes, the number of distance classes (i.e. the maximum distance), whether to consider an isotropic model or not and whether to consider a nested model. For a given data set, the number of pairs is fixed, the various decisions pertaining to the experimental variogram only determine how to apportion the pairs to the distance classes, angle windows. For a given application there is no theoretical assurance that if data were collected at a different set of locations that the apparent information in the experimental variogram would be the same. After all the data is only a finite subset (non-randomly selected) from one realization of the random function..

As noted or implied by several respondents there is a difference between using statistics that purport to quantify how well the model fits the experimental variogram and how well the kriging estimator (using that model) predicts/estimates values at non data locations. The former is only a means to the latter but not perfect.

The various cross validation statistics are useful for exploratory analysis as well as for examining how well the variogram "fits" the data. In particular those statistics may indicate that basic assumptions such as stationarity are not satisfied. The use should keep the ultimate use of the results in mind, i.e are they really useful or is it merely a mathematical/statistical exercise.

Most software packages that provide cross validation statistics use a moving search neighborhood with ordinary kriging, the choices pertaining to the search neighborhood can significantly affect the values of the cross validation statistics (e.g.,minimum and maximum number of data locations used, orientation of the search neighborhood, geographic size of the search neighborhood)

With respect to least squares fitting of a model to the experimental variogram, there are different possible weighting schemes and no one is best for every probem. Some of the statistics mentioned in other responses are those common in regression analysis which is based on an entirely different set of statistical assumptions.

Sorry, I notice a mistake in my earlier answer. The last sentence should read: If more than 1, the standard deviation of the kriging prediction error are too low and vice versa.

RMSE is a good criterion to decide of the most precise model among a series of tested models. ME is usually not very useful as all models give values close to 0 when the stationarity hypothesis is reasonable for the studied phenomenon. Finally, MSDR is useful only to appreciate if the kriging variance predicts correctly the precision of the kriging. If the value is too far from one (below or above), then the parameters of the model need to be adjusted. Note that when a model gives a MDSR say of "a", multiplying the Nugget and Sill of the variogram by "a" and rerunning the cross-validation will give a new MDSR of exactly one, whatever the model considered. So MDSR is not a statistic to choose between models.

When cross-validation is done, it is important to try to reproduce estimation situation corresponding to the typical kriging layout. Hence, if data are sampled along lines, it is necessary to decimate the samples along the lines to increase the distance between the samples and the cross-validated point. The larger distances will be more akin to the estimation of a point between the sampled lines.

Also, cross-validation can be used to decide of the neighborhood to use for kriging. Sometimes this decision is even more critical than the choice of the model.

You might have a look at:

Marcotte D., 1995. Generalized cross-validation for covariance model selection and parameter estimation. Math Geol., 27, 6, 749-762.

I would suggest MAE and MBE to select the optimal model.

please go to :

http://en.wikipedia.org/wiki/Mean_absolute_error

Have you compared the error of different interpolation methods?

It is one of the criteria for selecting models in geo statistics. It may not necessarily give the optimal.

To Jian

There is no way to determine the "error" for interpolation methods since that requires knowing the true values in which case there would be no need to interpolate

As Eghwerido has mentioned, cross validation is one tool for evaluating how well a variogram model fits a data set but the cross validation statistics are also affected by the parameters of the search neighborhood.

It is also possible to split the data set, i.e. the set of data locations, into a modeling set and a validation set but that still leaves open the question of whether the statistical assumptions and variogram model are satisfactory for both data sets.

There is no assurance that choosing the variogram model with the three statistics mentioned by Mooskood will produce a single unique choice, in particular they are not equally sensitive to changes in the model type (Spherical, Gaussian, Exponential, etc) nor to changes in the model parameters,

Predictive performances of the fitted models are checked on the basis of cross validation tests. The values of coefficient of determination R2, mean error (ME), mean square error (MSE), reduced mean error (RME), and reduced mean square error (RMSE) are estimate to ascertain the performance of the developed models.

Boutrldjaoui

Your statement is all wrong. First of all there are no "cross validation tests". Secondly they do not check the "predictive performance" of the fitted models. None of the three statistics you mention are relevant to cross validation, those pertain to regression and the statistical assumptions for regression and kriging are completely different.