Here my answer to your questions:

Question 1.) I am interested in developing a split-sample technique to assess the internal validation of a logistic regression.

You should not develop your own split-samling technique. There are good solutions.

If you have many data available than split you data randomly in a training (2/3) and validation set (1/3). Develop your model on the training set and validate your model on the validation set. Calculate on the validation set statsitics to quantify the goodnes of your model. There are many statsitics to use to quantify the goodness of the model.

If you do logistic regression you should use a ROC and its AUC. You also could use instruments like the PIT, the Brier Score or Scoring rules. There is no clear guidance which are the best. Use a selection of them and present the results to have a check if they behave consistently.

In case you do a Cox-regression the problem of quantifying the (predictive) quality of your model is technically a bit more difficult.

If you have not a lot of data you should do a cross-validation approach which may be complicated and nested if you have to tune a lot of parameters. As literature I would recommend:

Ruschhaupt M, Huber W, Poustka A, Mansmann U. (2004) A compendium to ensure computational reproducibility in high-dimensional classification tasks. Stat Appl Genet Mol Biol. 2004;3:Article37.

Do not be confused by the word high-dimensional contained in the title.

Certainly, the work (especially the book) of Ewout Steyerberg offers a good guidance.

Question 2.) Many statistics have been used to compare both groups, training sample and validation sample: MSE, deviance residuals, AUC, R2, adjusted R2...

I do not understand this questions, what are "the two groups". Certainly there is no comparison between the results of the results of the training and the validation sample.

It is obvious that therer will be a difference.

Hi,

It is enough with a simple validation process. Personaly, I prefer the leave-one-out, it is replicable and does not depend on the "random" selection of the partitions. However, other cross-validation procedure are also valid and must be achieve similar conclusion.

The parameter to check, in a logistic regressio....n the AUC is a good parameter but you also can compute an automatic threshold (for instance the related with the Youden Index one) and reported the obtained sensitivity, specificity, PPV and NPV.

I think the method described by EW Steyerberg et al using bootstrap is good as it gives you the optimism corrected value for AUC, Rsquare etc.. and also uses all the cases at each run of the bootstrap.

I'm not sure to understand: normally cross-validation is a tool to compare two models, not to compare two subsamples. In the first paper you cite, the conclusion is that split-sample techniques are worse than cross-validation, so I would rather use the latter and use a proper scoring rule (such as Kullback-Leibler) to assess the predictive power of each regression model.

For logistic regression, I would prefer the AUC, if you do not attempt to choose a cut-off value or misclassification error if you do. Both parameters are not very sensitive in that different models often have similar performance measured by them. The log-likelihood function itself (after CV) could be more sensitive but its interpretation is not so straightforward. For AUC, you need to do at least leave-two-out cross-validation.

Maybe you can also consider Durbin-Watson statistics on autocorrelation of residual values, especially for training. The deviation and mean of residuals do not have the same information content.

It depends on your objectives.

Are you interested in prediction of future examples, or on inferences about the parameters? Or scientific insight (sort of like inference)? The simple case is prediction for future (to be obtained) examples. Then leave-one-out (as advocated by Pablo Martínez-Camblor above) is probably best. The other stuff depends on your conceptualization (i.e., your model) of the phenomenon.

The re sampling method you apply (bootstrapping, bagging, n-fold cross validating) depends quite a bit on the size of your dataset. When you have a small dataset, bootstrapping or leave one out normally works better than n-fold cross validation. Although the bootstrapping has the disadvantage of randomness in the results, this should be negligible when you repeat sufficiently.

Which statistic to look at indeed depends on the purpose of your study. AUC is nice because you don't have to specify a threshold, but it doesn't really give an indication of accuracy, but more an indication of "discriminative power" of the model. When you have many occurrences of one of the possible outcomes in the logistic regression (assuming a binomial response) AUC tends to be low in general.

Other statistics to consider for this are Cohen's Kappa (or Maximum Kappa,where you threshold you data at the point where Kappa is maximum) or the True Skill Statistic (TSS). A nice reference for these kind of statistics is:

OMRI ALLOUCHE, ASAF TSOAR and RONEN KADMON (2006) Assessing the accuracy of species distribution models: prevalence, kappa and the true skill statistic (TSS) Journal of Applied Ecology 2006 43, 1223–1232

Here my answer to your questions:

Question 1.) I am interested in developing a split-sample technique to assess the internal validation of a logistic regression.

You should not develop your own split-samling technique. There are good solutions.

If you have many data available than split you data randomly in a training (2/3) and validation set (1/3). Develop your model on the training set and validate your model on the validation set. Calculate on the validation set statsitics to quantify the goodnes of your model. There are many statsitics to use to quantify the goodness of the model.

If you do logistic regression you should use a ROC and its AUC. You also could use instruments like the PIT, the Brier Score or Scoring rules. There is no clear guidance which are the best. Use a selection of them and present the results to have a check if they behave consistently.

In case you do a Cox-regression the problem of quantifying the (predictive) quality of your model is technically a bit more difficult.

If you have not a lot of data you should do a cross-validation approach which may be complicated and nested if you have to tune a lot of parameters. As literature I would recommend:

Ruschhaupt M, Huber W, Poustka A, Mansmann U. (2004) A compendium to ensure computational reproducibility in high-dimensional classification tasks. Stat Appl Genet Mol Biol. 2004;3:Article37.

Do not be confused by the word high-dimensional contained in the title.

Certainly, the work (especially the book) of Ewout Steyerberg offers a good guidance.

Question 2.) Many statistics have been used to compare both groups, training sample and validation sample: MSE, deviance residuals, AUC, R2, adjusted R2...

I do not understand this questions, what are "the two groups". Certainly there is no comparison between the results of the results of the training and the validation sample.

It is obvious that therer will be a difference.

I agreee with Ulrich Mansmann. But you might also consider sensitivity, specificity and Nagelerke's Rsquare in your validation set if you do logistic regression.But in that case PPV and NPV might be misleading if you have more controls than cases (like 2:1 ratio). In R its quite simple to split your data, lets say 1000 times in training and validation sets. So you get an idea of the distribution of the statistics your interested in.

Thanks a lot for your comments and for your time.

My sample is composed by 93 cases and 414 unmatched controls and a logistic regression has been developed. In a first approach, the validation of the model was initially conducted as follows:

1. The predictive performance of the logistic regression: ROC curve and AUC (discrimination), Hosmer-Lemeshow test (calibration), and Classification table (goodness of fit).

2. The cross-validation, focused to the internal validity. Through the 50/50 split sample technique, 4 iterations, with deviance residuals and R2 as parameters of interest:

DEVIANCE RESIDUALS *

Estimation data (N=253) / Prediction data (N=254)

Iteration 1: - 0.15 (0.9) / - 0.16 (0.9)

Iteration 2: - 0.15 (0.9) / - 0.16 (0.8)

Iteration 3: - 0.16 (0.9) / - 0.14 (0.9)

Iteration 4: - 0.14 (0.9) / - 0.16 (0.8)

* Mean (SD)

R² NAGELKERKE

Estimation data (N=253) / Prediction data (N=254)

Iteration 1: 0.22 / 0.17

Iteration 2: 0.17 / 0.20

Iteration 3: 0.17 / 0.25

Iteration 4: 0.19 / 0.22

Although the results estimated on the first data predict reasonably the second set of data, it was here where I asked for another opinion.

A) It might be interesting to know how many independent variables you got.

B) Did you use a forward or backward logistic regression analysis? If your aim is prediction and you got many variables the backward might be more appropriate.

C) The 50/50 split technique might not be the best choice. It might be better to apply a 1:1 or 1:2 case control split for validation sets since it might happen that u end with a 1:6 ratio if you just select a random sample (250 observation at all). You could take a random sample from cases (n=60) and a random sample from controls (n=300 or n=350) as training data and the remaining for validation where the case /control ratio would be around 1:2. Even though the 1:1 design would best but I know it is mostly very difficult to get new cases and using all the data is preferable.

D) If I am not wrong you did this split (iteration) only 4 times this should be done much more times.

You can use the sensitivity and specificity in your validation sample to show how well your model perform.

Great multidisciplinary discussion. As always, I like to emphasize that the research question is central. I take it that the aim of the validation is to estimate the quality of the model in predicting an outcome. For epidemiological research, several of the more common performance measures are in Frank Harrell's val.prob function in his rms library for R, including AUC (c statistic), Brier score, Nagelkerke R2, etc. [Note: c is calculated from Somer's D as c=(D/2+0.5)].

Also, I like the calibration slope, which is a measure of how much shrinkage would be required to improve calibration of the predictions from the model. The shrinkage idea (see e.g. Copas 1983; and Van Houwelingen 1990) follows from a classical recalibration framework (Cox 1958).

It is important for internal validation that all modeling steps are considered. The val.prob function can readily deal with backward stepwise selection (repeated in every bootstrap replication), but not with other perhaps more subjective modeling decisions that were taken such as lumping of categories, choosing transformations to deal with non-linearity, etc. Such model uncertainty is a key factor to take into account at validation.

Thanks to others for mentioning my JCE 2001 paper on internal validation (PMID: 11470385, which confirms that split-sample is suboptimal and bootstrapping is preferred for the more common epidemiological research settings) , and my 2009 book (Springer). Both follow others who did the more fundamental work; this is also the case for a recent review of performance measures in prediction research (Epidemiology 2010, PMID: 20010215).