Hi all!
I am running some IRT models using the mirt package for R (Polytomous 2PL GRM). The scales I am analysing consist of 6-items that are comprised of 3 contrait / 3 protrait items measured on 9-point Likert scales, and a sample size is around 3200.
When I run the item-fit (S-X2) the output shows that the protrait items fit the model (items 3-5) and the contrait ones (items 6-8) do not (output below). Several of the other scales I am testing have similar issues but not always with the protrait/contrait divide. Assumption testing showed a very strong one factor solution and acceptable Local Dependence (LD) (Yen's Q pairs were around .02 - .25). Although many texts recommend the S-X2 test for polytomous item-level fit, none that I have found suggest what to do when the model does not fit the data (although several just say "close enough" which seems like troublesome logic).
item Zh S_X2 df p
1 ethnic_3 7.12 210.88 199 0.27
2 ethnic_4 8.33 199.63 192 0.34
3 ethnic_5 6.43 219.67 196 0.12
4 ethnic_6 5.31 286.77 211 0.00
5 ethnic_7 4.48 254.58 213 0.03
6 ethnic_8 1.87 308.25 234 0.00
I was wondering what levels of violation are acceptable and what we can do to increase the item-level fit? Is the only solution to change the model (1PL, 2PL, Partial Credit etc., ) or estimator used?
Thanks in advance,
Conal Monaghan
Dear Colan
item fit affected by many issues such as the number of ability intervals, some softwares use it equals to 10 as a default option, I would recommend you to try to change them to a less number such as 6 or 7 then you might have better results.
Regards
Hi Rana,
Thanks for the response. Are ability intervals the same as response categories or parameters? Thus one way to improve fit would be to average response options 1 and 2 (Strongly Disagree and Disagree) and options 6 and 7 (Agree and Strongly Agree)?
Conal
ability continuum in IRT which practically ranges [-3,3] divided to intervals, it is not the response categories, these intervals affect the item fit because the Df computation depends on them beside another variables, so when the ability continuum divided in a larger number of intervals the probability of misfit will increase.
Interesting, would you know how I would accomplish this? I am currently running the analysis in mirt() for r statistics
I am sorry I did not use mirt before, I made that easily using Bilog. But for sure if you google it you will find many publications and recommendations. I hope that I could use R it is so complicated, I would be appreciated if you provide me any help or papers how to use mirt!
Hi, Conal:
I think you are unlikely to find blanket recommendations for responding to significant S-X2 tests because while S-X2 can tell you that there is a problem, I don't believe that it can tell you why. For that, you have to interpret your other sources of information in the context of your theory.
Given that the approach to each significant S-X2 test is unique, I can't offer any guesses about the item sets you do not describe here. In the case of the items you describe in your question, my guess is that you may have a multi-dimensionality issue. This guess is based on two pieces of evidence. First, the Q3 cutoff of .2 is a little arbitrary, so sometimes S-X2 picks up problems that Q3 misses. Second, you mention that the first three items are "pro" and the second three are "con." There's a fair amount of literature suggesting that negatively-worded or negatively-valenced items can form separate factors from their positive counterparts, even though they theoretically shouldn't. This "method effect" can often be controlled for by using a bifactor model.
If what I just said sounds correct, you might want to consider running a bi-factor model to account for the possibility of the method effect. I've seen this done in a couple of different ways, most typically with all the items loading onto one factor (i.e., items 3-8 would load onto one factor, and that would be your content factor), and the negatively-worded items loading onto a second factor (i.e., items 6-8 would also load onto a second factor, and that would be your negative-item-wording method-effect factor). An example: doi.org/10.1037/a0036472
The second--and less common--way I've seen this done is to also model a third factor to account for the positively-worded items (i.e., items 3-5 would also load onto a third factor, and that would be your positive-item-wording method-effect factor). An example: doi:10.1016/j.paid.2014.03.034
Unfortunately, I am not familiar with mirt(), so can't tell you how it would be programmed.
Best.
Hi Nicole,
Thank you for your help with this! I agree that the issue is most likely a method issue. It is also present in many of the other subscales we tested (with one subscale only having 1 out of 6 items fitting well which is scary ;/ ). The paper you sent through (doi.org/10.1037/a0036472) is a great example of how this effect occurs and potential solutions, thanks for that. I will try to solve this problem in R through mirt() and post it on here if we solve it.
I wonder then, how this would affect interpretation of the results? can we interpret it the same way or is it now different due to the method factor?
Cheers and thanks again,
Conal
- Badges
- Science topic
- Similar topics
- Correction