It is enough to show that the set of predictors held up, and therefore are valid. It is not enough to validate the equation if you are proposing that certain cutoff scores carry certain meanings (for example, if you are predicting success in school defined as a 2.00 GPA). Beta weights are notoriously unstable, by the way, especially in multiple regression equations with many predictors. So in that sense you have both validated and failed to validate the regression model. You should report both the success (i.e., the strong correlation between predicted and obtained values) and the failure (i.e., the fact that the equation cannot be relied on to produce exact point estimates).

I have two other thoughts to offer. First, it makes a big difference how you originally derived the equation, how strongly it correlated with outcome in the original sample, and how many predictors you have. If (say) you are using a small set of predictors (two or three) that were derived based on a theoretical model, then your findings are more impressive. And if the original correlation wasn't much stronger than the replication one (say, r = .70) then your model suffered little validity shrinkage and is all the more impressive. On the other hand, if you used a scattershot statistical technique (such as stepwise regression) and derived a model with many predictors (say, ten or more), then the results are much less impressive - odds are that some of your "predictors" were the product of chance alone.

Second, if your correlation of .60 is only significant at p = .034 or thereabouts, it sounds like you have a very, very small sample (N < 15). In general, we are told not to use multiple regression with such small samples. But maybe this was just a slip of the keyboard.

Thank you for your text, Stephen. I think that I am right in the middle. I will explain the data: 4 predictors, three from a test and one from other test. Criterion is a continous variable without cutoff. The original equation was obtained after I split a sample n=19 into two halves. R = .977. All the four predictors are significant even after controlling for depression and BMI.

Beta coefficients from this sample are used to predict the criterion in a sample n=17, and the correlation between Y and Y' = .590, p=.013 (not .60, p=.034). For mean differences between Y (7.07) and Y' (15.14), p = .000

However, Mean BMI in the second sample is lower, indicating a more severe illness. Maybe the influence of BMI on cognition is not linear, and extreme lower BMI are associated with worse performance and as BMI increases cognition plateau.

I think your main problem is that the sample is basically too small. The results should probably be viewed as exploratory, a kind of pilot study. They are promising, because the multiple R is so strong and you don't have so many predictors that there's a concern about "overfitting" the model. Worth reporting at a conference and worth following up on. But you may have trouble getting anything published with only 36 in a correlational study.

Javier, to me the exercise of correlating the predicted and raw scores is rather circular, since regression is essentially an expansion of the correlation approach. The split sample replication is by far more interesting, but is going to mostly cause confusion if the sample is small and almost invariably replicate the results if the sample is large. It is most useful if you are using a dangerous technique that has a lot of underlying testing, such as stepwise regression, but with very few exceptions, such as propensity scoring, I don't know why anyone would do that. I think that all the standard ways of evaluating a regression, such as the F-statistic, particularly to compare nested models as you add variables, t-tests for individual coefficients, overall r-square or adjusted r-square are the ways you confirm your regression. Furthermore, it is worth testing regression assumptions, which largely pertain to the residuals. Bob

Hi Robert. Thank you a lot for your reply. Maybe I didn't explain correctly the procedure. I correlated raw and predictd scores in another sample different from that with which the equation was built. So, I can't see the point in circularity. I would see it if I correlated raw and predicted scores in the same sample, isn't it? I evaluated the regression with all the ways you referred to, but I want to test if that equation is useful for other samples. I too conducted analyses to calculate the relative weight and commonality, to improve information about the regression, but I would like to know how to interpret that this equation is useful. Double crossed validation, maybe? Thanks a lot again.

Yes, Javier, I did not realize the correlation was one of the things you did in the separate population. It is certainly helpful to see if you model holds up in that way. Ultimately, replication is a big part of how we come to accept scientific findings. I cannot remember the details, but a surprising number of results even in the highest impact journals are never replicated. Bob

Thanks Bob, I agree with your last sentence. But that doesn't seem to be enough to get a paper published (unless you are very famous, I guess).

Here's what I did:

- Step one: split the sample (N = 24) and got a regression equation in one half (n=12). R2 = .872, p = .003

- Step two: predicted scores on the second half (n=12) and correlated raw (Y) and predicted (Y') scores. Ry-y' = .801.

- Step three: added 12 additional participants (n=24) to step 2 and predicted scores on the criterion with the equation from step 1. Then I correlated raw and predicted scores. rRy-y' = .789.

Additionally, I calculated structure coefficients, commonality weights and relative importance weights for predictor variables in Step 1.

So you think that's correct?

As an exploratory analysis, I think it is interesting enough to get published. I hope that reviewers and editors see it the same way :)

Thank you all for your help and suggestions.

Bob, I forgot. A 2x2 ANOVA with repeated measures in one factor (factor Y, but not factor group) revealed that there were no differences in fator Y, Factor group nor interaction.

The procedure you are referring to is called cross-validation. Basically you split your sample into two different groups, a validation group and a cross validation group. You develop a model in one sample by whatever means (multiple regression, voodoo, or whatever) and then ascertain how well it fits by applying the model to the independent cross-validation group. The correlation between the observed values and fitted values in the cross-validation sample is a nearly unbiased estimate of how well the model will work in the population.

Extending the cross-validation idea to very small samples where a traditional cross-validation cannot be performed because the sample is too small have also been proposed and tested. If your total sample is not too large, then you may be able to manage the process by hand. In this procedure you would look at all (n-1) and 1 splits.

remove one observation (person) and develop a regression equation on the remaining n-1 subjects. Apply your regression to the one subject previously left out and get a predicted y value for that subject. Put the subject back into the sample and remove the next subject. Repeat the procedure of developing a regressin equation on the n-1 sample and apply it to the one left out. Do this for eaach subject in your sample.

The resulting correlation between the predicted scores and the observed scores is an estimate of how well a regression equation calculated on all n subjects will work in the population . Since each of the equations developed in the n-1 subjects is not independent of each other, there is going to be a slight bias tn the final estimate of how well the regression equation will work in the population from which you sample is drawn.

For more information, see

Drehmer, D. E. & Morris, G. W. Cross Validation with Small Samples: An Algorithm for Computing Gollob's Estimator. Educational and Psychological Measurement, 1981, 41, 195-201.

There are also a variety of statistical estimators based on correction to the degrees of freedom, standard errors, etc. and are summarized in

Cotter, K. L. & Raju, N. S. (1982). An evaluation of formula-based population squared cross-validity estimates and factor score estimates in prediction. Educational and Psychological Measurement, 42, 493-516

I hope that you find this helpful.

Maybe I got that wrong, but to me it sound like a cumbersome and inaccurate, even if more easily understandable, way of making things. Bootstrapping procedures exists for numerous statistic parameters estimating techniques and are implemented even in “canned” statistical software (like SPSS and similar). I guess that, in your case, bootstrapping is the way to go. Cross-validation is, in my opinion, useful for two things. Validating models from previous (emphasis on previous) researches and eyeballing the stability of the model (due to the fact that you can easily compare some parameters: R^2 and such).

Hope it helps.

Regards,

Marko

I would recommend bootstrapping the entire process of splitting the sample in half and testing the equation on the other half given the very small sample size. The p-values and standard errors are questionable and the bootstrap would allow you to generate empirical p-values and standard errors for the coefficients and the r-squared values. Bootstrapping the process will give you a better sense of the magnitude of the uncertainty in the solution than relying on the theory based standard errors and p-values. If you don't want to bootstrap, consider generating an exact p-value by taking all possible samples of size 12 without replacement and cross validating in the alternate samples of size 12. Either of these processes will allow you to place trustworthy bounds on the cross validated correlation.