Hello Brendan,

The first question is, do you intend to compare each of the 177 items (ouch!) pre- and post, or do you have a smaller number of subscores (or total scores) from the instrument that will be the focus of your retest stability inquiry?

If the individual items are to be compared, then either a simple proportion agreement index (the proportion of respondents whose responses were the same on both pre- and post- occasions), or Cohen's Kappa (proportion agreement corrected for chance-level agreement) will work. In SPSS, just be sure each item and each occasion is entered as separate columns in your data worksheet.

If subscores or total scores are the outcome of interest, then you can either directly enter these as individual columns in the data worksheet or have SPSS compute them via syntax: compute pre_total = item1_pre + item2_pre + .... (etc.).

Then, correlate the summated pretest score with its corresponding posttest score. Higher correlation indicates more stability over time.

Good luck with your work.

Use the following resources:

(a) Pallant, J. (2016) SPSS Survival Manual: A Step by Step Guide to Data Analysis Using the SPSS Program (6th ed.). Sydney, Australia: Allen & Unwin.

(b) Field, A. P. (2013) Discovering Statistics using SPSS (4th ed.). London: Sage.

(c) Bryman, A. and Cramer, D. (2012) Quantitative Data Analysis with IBM SPSS 17, 18 & 19: A Guide for Social Scientists. London: Routledge.

(d) https://www.youtube.com/watch?v=fmF0jZtZCRQ

(e) https://www.youtube.com/watch?v=L-KXfBIn4KQ

Follow.

Hello Brendan,

As David Morse has indicated above, please make sure that you enter your data such that each variable is in a separate column. That means that each participant should have only one row. And, as I hear David saying, analysing all 177 variables separately is an "ouch" setup. I strongly recommend that you analyse at the (valid) subscale level, not at the individual item level.

I suggest that, for your 177 variables you name them in an easy way, for example, 1V1, 1V2, 1V3 . . . 1V177 for the Time 1 administration, and 2V1, 2V2, 2V3 . . . 2V177 for the Time 2 administration.

I also recommend that you consider analysing your data with intraclass correlation coefficients in addition to, or perhaps rather than, Pearson product-moment correlations.

Intraclass correlations are pretty easy to do in SPSS, but you need to make the right choices lest you obtain inflated outcomes. In particular, I am pretty sure that you should not choose the averaged output.

Robt.

Thank you David Morse and Robert Trevethan for your answers!

I don't think I will compare each of the individual items, but I will narrow it down to the most important ones. With that being said, I have an ordinal response scale that ranges from 1 - 5. I read that for these measures, a weighted kappa should be used to calculate test-retest reliability. Can you guys confirm if this is correct?

I also think I will use intraclass correlation coefficients for the subtests, but this has turned out to be slightly more complex than anticipated.

Hello Brendan,

As a first reaction, I would recommend against basing test-retest analyses on single items, so I wouldn't get into using kappa.

Instead, I would try to work with totals for subscales. David M. might recommend a different strategy, however - and I'm happy for him to provide different advice if he would like to do so.

Don't be too perturbed by intraclass correlation coefficients (ICCs). They are really quite easy to obtain in SPSS as long as you press the right buttons.

I have written an article about ICCs that you might find helpful. Here's the reference for it:

Trevethan, R. (2017). Intraclass correlation coefficients: Clearing the air, extending some cautions, and making some requests. Health Services and Outcomes Research Methodology, 17, 127–143. https://doi.org/10.1007/s10742-016-0156-6

There is a pre-publication version on ResearchGate that you can find by googling "Intraclass correlation coefficient Trevethan".

There is another article, by Koo and My, that you might find helpful.: doi:10.1016/j.jcm.2016.02.012

Feel free to get back if I might be able to help more. In SPSS, essentially, it's Scale / Reliability Analysis / Enter your scales into the items box / Statistics. Under Model, leave/choose the first/default option (2-Way Mixed) / under Type, leave/choose the first/default option (Absolute Agreement) / Continue / OK. Then with your output use the output that refers to Single Measures. If you use Average Measures, you will be working with a spuriously inflated ICC - something that quite a few researchers do in my experience, largely because they want their results to appear as impressive as possible even if doing so is invalid / dishonest.

You should report your kind of ICC, the actual ICC value, and the 95% confidence intervals. I think you can ignore the stats relating to F.

Robt.

Hi Robert Trevethan ,

I was able to run the ICCs with no difficulties. Thank you so much for your help and guidance!

Hello Brendan,

Thanks for the feedback. I'm really pleased for you: Congratulations!

Robt.