If you have absolute measures to correlate each with, you could just correlate each of the measures with that absolute and select the one with the highest correlation. Another approach would be to do the measurements with both measures, take the differences, and hypothesize the mean difference is zero.  If non-significant, it should not matter which you use.  It very well might be non-significant as your correlation would suggest that over 76% of the variances of the two measures is in common.  By the way, you could use a parametric or non-parametric test of that hypothesis depending on satisfaction of the appropriate assumptions.

Thank you for your answer. For completeness, both the scores are expressed in absolute values with discrete intervals (pirani from 0 to 6: interval .5; dimeglio from 0 to 20: interval 1) the rho coefficient was .524 (p

Dear Giovanni!

I was a bit puzzled about your question, but then I figured out that I have the problem with the specific question "but how to determine which is better in terms of predictability" What do you mean with better?

In a very simple way you could look at the two correlation coefficients descriptively and answer your question by yourself: the test with the higher correlation with the criterion is better.

Another way, but still simple, you could transform the correlation coefficients into Fisher's Z scores. With these Z scores you are able to calculate confidence intervals for each r, which helps to estimate how precise your estimation is and how much the two tests overlap. And you are also able to calculate, if the difference between these two coefficiants is significant (although this also depends on your sample size and will not give much additional information)

Another approach would be to use a blockwise multiple regression, to estimate the incremental validity: how much more variance is explained by the other test? How much do they have in common and how much is unique variance/information (but I don't know if this would be possible for you, since you only assume ordinal scale level).

But I think you have to take into account several other things beside the bivariate/multivariate correlation/regression. How reliable are the two measurement tools? Since this heavily influences the validity of your measures, i.e. the relationship between predictor and criterion. Have a look at Spearman's "correction for attenuation" formula to calculate the validities. Since the unreliability of your measurements may cover the true validity and reveal other effect sizes than the simple bivariate correlations.

Another point you should consider to decide which test is better is ecology. How many items has each test? If both test tend to have the same (corrected) validity, I would chose the shorter one or at least the one which is easier/cheaper to apply. This is also a question incremental validity: what information gain do I have at which cost? If I only explain for example 1% of incremental variance at the cost of a very complex and expensive procedure, this is probably not the way to got.

I hope I gave you some additional aspects to consider and food for thought, if you want to determine which test is "better"

Cheerz,

Rainer

sorry but far away from my competence dr. gluud denmark should know that

Cochran test

Dear Mahmoud,

do you mean Cochran's Q test? Please explain why and how this test?!

Best,

Rainer

Hi Rainer,

thank you for your answer, you clearly identified my problem. from an ecological point of view there are few differences between the two scores (6 items versus 8 items) and both the scores showed satisfactory reliability in termes of inter-rater agreement. I tried with the Fischer's Z score finding a p-value of .06 between the two correlation coefficients (that is not significant for the size of population in study). In contrast blockwise multiple regression showed that the dimeglio (B coeff. =.301; Sign. = .000) was more predictive than pirani (B coeff. =.-071; Sign. = .678). So now my problem is: which is the statistical test that gives me, the more realistic answer?

So, let me recapitulate your results:

1) The dimeglio scale shows a higher bivariate correlation with your criterion

2) The Z-Test indicates a nearly significant difference between the correlation coeffcients (which shouldn't be emphasized too much, but see 3) )

3) Although both have similar correlation coefficients, in the multiple regression analysis, the dimeglio scale shows a significant B weight, whereas the pirani scale doesn't, indicating that the pirani scale explains no unique variance.

4) both scale show a similar ecology (6 vs 8 items)

5) both show satisfactory reliabilities.

So, which one would you choose when you read this? I think I would stick to the dimeglio scale so far. But what you could do is to calculate the internal consistency Cronbach's alpha additionally to your inter rater reliability. After that I would consider to use Spearman's "correction for attenuation" to estimate the "true" validities of the scales.

Maybe this will point into another direction, but I would not expect that. But just do it, to put your decision on solid ground. If all indices point in the same direction, the decisiion will be pretty obvious.

Thank You Rainer.

All The Best.

Giovanni

You are welcome!

Good luck!

Rainer