Hello Lili,

You didn't indicate what the number of items or the sample size was for your data set, or whether you're using a dichotomous or polytomous model, so I'll be a bit general.

The Martin-Löf test can be influenced by: (a) number of items; (b) sample size; (c) how the items are split, even when unidimensionality is present (see: https://www.dgps.de/fachgruppen/methoden/mpr-online/issue9/art4/Verguts.htm).

If there was a larger set of misfitting person-item outcomes in one item split than in the other, that could influence the results. Ditto for unequal appearances of local dependence in person-item outcomes. Ideally, a radically different assortment of item difficulties across splits shouldn't trigger false positive results, but possibly it could.

Of course, multidimensionality is always a possibility. I don't recall having seen a study evaluating how dissimilar (absolute values of correlations different from 1) the dimensions have to be for any specific test to have good power.

I'd suggest that one place to start would be evaluating person- and item-fit indices within the item split sets and see if that helps in any way.

Good luck with your work.

Dear Lili,

I can only second what David and Georg already said. Other than that I would not recommend using a median split for the Andersen LRT. Simulation studies show that the Andersen LRT performs rather poorly with a random split (https://dx.doi.org/10.22237/jmasm/1555594442).

All the best for your research,

Georg

Dear Lili,

I am not completely sure whether this question is still pending? David and Georg and Georg have already commented on it.

Georg Hosoya

"Since the fundamental paper of Guttman [1954] on the simplex structure for correlations between ordered tests, many investigators have found data displaying the typical simplex structure. Guttman gave several examples of this structure. His Table 5 is reproduced here as Table 7. In this example all the tests involve verbal ability and are ordered according to increasing complexity.

The typical property in a simplex correlation structure, such as that in Table 7, is that the correlations decrease as one moves away from the main diagonal. Such data will not usually fit a factor analysis model with one common factor for the following reasons. Let the tests be ordered so that the factor loadings decrease. Then if the factor model holds, the correlations in the first row decrease as one moves away from the diagonal, but the correlations in the last row increase as one moves away from the diagonal. Also the correlations just below the diagonal decrease markedly as one moves down. These features do not hold in Table 7 " (Jörgskog, 1978:, page 469).

I am not sure whether modern applications of factor analysis have resolved this. I myself have not found indications it did and the few empirical tests I performed myself suggest that the observation by Jörgskog is still valid.

The above have led me to develop an understanding that unidimensionality has no singular shape. With this in mind, I caution against shifting between distinct psychometric traditions.

Lili Poki I completely agree more David Morse comments about numbers of items and sample size.

I would like to add to his comments that, as I understand it, the assumption of Local Independence (LI), applies to the LR-test and the most clean way is to assess LI before assessing the LR-test itself. I think of Ponocny's non-parameteric test as a good solution for this, but Ponocny's tests are restricted to cases of dichotomous responses. Note that I always correct LI-tests for chance inflation (Koller & Hatzinger, 2013). I have also aplied Chen and Thissed LD statistic and which, if corrected for chance inflation, gives very similar results to the Ponocny's test and which is, as I recall, available for polytomous models.

Furthermore, if data has nested or clustered groups of persons, then I would advise to re-perform the LR-test for the hypothetical "unclustered" situation. A quick way to do this is by randomly selecting one person from each group. Off course there are possibilities to perform a multilevel LR-test, but this is very computational intensive and for polytomous models I am only aware of Baysian applications applications. The work I know about in this area only gives technical and indirect descriptions about how you might perform this test (e.g., Bürkner, 2019). I usually stick with the random selections.

I wish you all the best and hope that this might help.

With kind regards,

Rikkert

Bürkner, P. C. (2019). Bayesian Item Response Modeling in R with brms and Stan. arXiv preprint arXiv:1905.09501.

Jöreskog, K. G. (1978). Structural analysis of covariance and correlation matrices. Psychometrika, 43(4), 443-477.

Koller, I. & Hatzinger R. (2013). Nonparametric tests for the Rasch model: explanation, development, and application of quasi-exact tests for small samples. Interstat, 11, 1-16.