Dear colleagues,
I have encountered a statistical issue involving the analysis of Likert scale data, specifically using the POSAS scare scale, in a paired situation. The scale comprises numerous questions, and I am considering two approaches for comparison: either evaluating each item separately between two groups or aggregating the values to obtain a final score for subsequent comparison.
However, a challenge arises when I choose to add up the scores. In such cases, obtaining significant results does not provide clarity on where the differences originated. Conversely, if I opt to compare each item individually, the number of tests would escalate to nearly 80, making it sensible to consider Bonferroni correction. Nevertheless, dividing the significance threshold (0.05) by 80 renders many scores non-significant.
I am contemplating whether it is reasonable to adjust with Bonferroni correction by dividing by the total number of tests. Alternatively, I am considering adjusting for each Likert item separately. For instance, in a two-time point comparison, dividing by 2 for each Likert item seems more appropriate than aggregating all Likert scale comparisons, which would result in a division by 70-80 tests.
I would greatly appreciate your assistance and insights on this matter.
Hello Farzan,
The answer depends on two things: (a) your specific research question; and (b) the structure of the POSAS instrument.
If your research question addresses whether groups have equal score distributions on each and every item/stimulus/question on the POSAS, then you'll need to compare on all 80, and use a suitable p-value adjustment method. Several are better than the Bonferroni (though it is conceptually and practically very straightforward), such as Holm-Bonferroni or Benjamini-Hochberg. See R documentation on p-value adjustment options at:
https://rdocumentation.org/packages/stats/versions/3.6.2/topics/p.adjust
If your research question addresses whether groups have identical distributions on one or more of the constructs measured by the POSAS, then you'll need to determine how many such constructs POSAS covers, which items associate with each construct, and form those scores appropriately, then run the implied set of k comparisons (where k is the number of constructs).
This might be as simple as referring to a technical manual for the measure (if one exists), previous literature on the internal structure of the measure (if such exists, using samples relevant to your target population), or running your own factor analysis of the instrument.
Good luck with your work.
Dear Dr. David Morse
Thank you for your suggestions. Each POSAS Likert item assesses a specific feature of the scar, such as pigmentation, vascularity, etc. The total number of items is 16. I want to gain a general understanding of this problem. The question is, why do I have to consider all hypotheses at the same time for adjusting the p-value? What does the p-value of vascularity have to do with pigmentation?
Hello again Farzan,
The general issue is that of aggregate risk of type I error. The more hypotheses you test, the greater the likelihood of committing one or more type I errors.
I understand that you might consider scar features such as vascularity and pigment to be independent. However, you are, via the instrument, asking the same persons for multiple judgments and those judgments may not be independent. However, that's really not the central issue. Instead, it's the the sheer number of statistical comparisons you intend to conduct in the study.
You always have the choice as to how to set your decision rule (e.g., whatever p-value criterion for statistical significance you like). However, you'll tend to find that reviewers might be wary of studies with many significance tests in which p-value adjustments are not made.
Good luck with your work.
David Morse Thank you for your valuable insights.
One last question has crossed my mind: Can I split the tests based on dependence? For instance, if I want to run 14 and 28 comparisons based on the observer and patient parts of the questionnaire, which have 7 and 14 questions respectively, is it acceptable to divide p-values by 14, and for the patient part, by 28? Does splitting make sense in this context?
Hello again Farzan,
Going this way is akin to considering each batch (observer set, patient set) as a "family" of tests, and adjusting type I error thresholds separately by batch. Yes, you could do just that. Just explain your method in the study write-up.
Good luck with your work.
- Badges
- Science topic
- Similar topics
- Computer Science
- Computer Communications (Networks)
- Internet