I'm hoping to get some help choosing the appropriate statistical analysis of Likert scale data collected from a pre- and post-intervention survey. These questions asked participants to rate their perceived skill level with various competencies on a scale of 0 (no knowledge - 6 (very knowledgeable). The same survey was used each time and the sample was the same group of 8 participants. The survey was conducted anonymously and unfortunately, reliable identifiers were not assigned to each participant. My question is, beyond simple things like descriptives and crosstabs to compare means, are there other statistical analysis that I could run? Would paired samples t tests be valid without having identifiers for participants since both variables have data from the same group and 100% response rate? Thank you for your help, this is a mixed-methods study and quantitative analysis is definitely my weaker area. I'm using SPSS for analysis if that matters.
Donavan Goff, just quickly, a paired-samples t-test is out of the question because you need to be able to match up the data for each participant at each timepoint.
Furthermore, if you are analysing one competency at a time, it might be more appropriate to use nonparametric statistics such as modes, medians, and interquartile ranges. However, some researchers, and under some circumstances, might be happy if you were to use means.
Maybe you could consider a Mann-Whitney U-test - but I confess I'm not 100% sure about that. Perhaps someone else could provide advice regarding that course of action.
Having only eight participants might present you with some problems, particularly with regard to generalizability.
Unless we identify the pair of pre and post responses for each of the study participants, possibly, we can't go any further than descriptives at each time point.
There's no way you can perform any kind of paired test (paired t, Wilcoxon signed-rank test, McNemar, and so). This is because there's no physical way for you to pair the pairs of responses.
You might want to perform an unpaired test (Mann-Whitney, ordinal regression, t-test), but you will be violating the assumptions of these tests, because someone's before response is not independent of their after response. For the Mann-Whitney test, Conover, Practical Nonparametric Statistics, calls this assumption "mutual independence between two samples". The upshot here is that the reported p-value will not be correct.
So practically, what should you do? It depends on your purpose. If you don't need to be too rigorous, you might perform the unpaired test of interest to you, and make note that you are treating the samples as independent, even though they are not.
The power is decreased because of not matching the data. I urge not presenting descriptive not without showing the precision of the estimates. Doing so often leads readers into thinking the estimates are more precise than then are. It may be worth re-doing the study.
Disclaimer: Given that your variable is ordinal with a relatively small number of categories, I would not be very comfortable with a paired t-test even if you knew the pairings.
Having said that...if your variable was truly a metric variable (rather than ordinal) and if you had a pretty good estimate (from other sources) of the correlation between baseline and follow-up scores, then you could estimate the result for a paired t-test. (I.e., you can compute the SE of the paired difference from the sample size, the two SDs and the correlation between the two measures.) I would plug in a range of r-values to see how the results are affected.
HTH.
- Badges
- Science topic
- Similar topics
- Medicine
- Physiology
- Biosignals
- Biomedical Signal Processing
- Statistical Analysis