Hi Daniel Groß

You need to work on the concept of "longitudinal measurement invariance". A sample article looks like this:

Article Longitudinal Measurement Invariance of Likert-Type Learning ...

Hi Daniel,

unless you plan to investigate latent means and their differences across cohorts and time, you don't need scalar invariance.

And yes, I would set up longitudinal multi-cohort model and first test the overall fit (optimally with a chi-square test) and then of the loadings. The first step is crucial and could result in a re-specification of the model; the second is in most cases no problem.

Then, I would test for longitudinal metric invariance which should work too.

Don't forget to estimate the cross-time error covariances.

Good luck,

--Holger

Daniel Groß, I fully appreciate your concern - particularly because I suspect that measurement invariance, desirable as it might be, is much easier to talk about than to achieve.

I found the following article, by David Funder, very refreshing:

https://funderstorms.wordpress.com/2020/01/31/misgivings-some-thoughts-about-measurement-invariance/

I am interested to see what advice you might be given.

Hi Robert,

I respect David Funder a lot and I surely agrees with regard to the prohibition aspect. MI is a psychometric feature of the measure as e.g. reliability and similarly, researchers should neither shot themselves nor abandon the study.

The whole point is that when your instrument does meet certain requirements you are on shaky grounds with regard to your substantive conclusion.

Apart from that, the whole piece is a collection of strawman arguments, problematic logic, and shows that he does not really know that MI is. The most obvious point is the comparison of correlation issue. You can only compare the correlation (or any other standardized measure) when both groups show similar variances/SDs. Hence if this is not the case, the actual effect may be equal the but the stand. coefficient differs. I understand that he does not like this fact (I don't either) but this does not change anything.

And of course, it can be debated whether the magical .05 border is reasonable (I agree more with qualitative interpretations of any threshold) but I find so many instances, where the significance level in MI was criticized by the same people who have no problems to speak of significant effects when they are in line with their goals :)

Hence, my stance is: test it (by a TEST and not delta CFI) and if significant, judge the extent of differences and the implications for the level of evidence of your substantive conclusions.

And of course, there a so many instance, where we cannot simply test for MI (for instance when we have single measures). Again this should not be a barrier to do the study but kept in mind as a limitation).

And finally: MI concerns the validity of a measure and whether you measure the same entity in both culture. Hence, it is very fundamental to analyse this as the unfortunate historical past with applying intelligence tests in African countries show. If you want to compare two groups, the instrument has to work properly and it has to measure the same thing. It is such simple.

All the best,

Holger

Holger Steinmetz, thank you for diving back in. I take on board (and agree with) many of the points you raise. Certainly, I'm sure that a lot of research is invalid because of a total failure to consider MI.

But I wonder whether there is a point beyond which trying to attain, or insisting on, MI is unreasonable. And, if there is that kind of point, I wonder where it lies. For example, if all kinds of MI don't need to be satisfied, which kinds do? And does any "cutoff point" apply in all studies where comparisons are involved?

I hope I'm not coming across as contentious. I'm possibly as puzzled by some of this as are many others. And I certainly don't want to ignore the issue.

Hi, you may take a look into my first published paper :)

Steinmetz, H., Schmidt, P., Tina-Booh, A., Wieczorek, S., & Schwartz, S. H. (2009). Testing measurement invariance using multigroup CFA: Differences between educational groups in human values measurement. Quality and Quantity, 43(4), 599-616. doi:10.1007/s11135-007-9143-x

You will find all forms not only explained but in addition, which forms are necessary for which substantive conclusion (see for instance Table 1 for a comprehensive list).

This is most important as researchers often test for things they don't need.

This is most obviously the case for scalar invariance that is necessary when you want to compare means of constructs. Again, if even PARTIAL scalar MI fails (which is another important relaxation) than you simply don't know whether the observed difference is due to a difference in the construct or the item response (e.g., differential response style).

In this simulation study, I made a blatantly simply simulation in which I simulated different item intercepts and analyzed whether a simple ANOVA type mean difference in the composite of all items would show a significant mean difference.

And wonder what happened ? ;)

Steinmetz, H. (2013). Analyzing observed composite differences across groups: Is partial measurement invariance enough? Methodology, 9(1), 1-12. doi:10.1027/1614-2241/a000049

If you want to test the difference between groups in the effect of a latent vairable, than all you need is configural and (partial) metric invariance. This is achiveable in 90% of the cases. Scalar invariance is difficult but that's how it is.

HTH

Holger

Holger Steinmetz, thanks for adding this. I'd love to be able to sit down with some actual results and talk through what was permissible and what not.

Given your most recent post, I wonder whether you might you add any advice that would help Daniel Groß

I am puzzled, I said him exactly what to do. What do you think is missing?

Holger Steinmetz, sorry for not being clearer. I was wondering, for example, whether you provided sufficient information about what you mean by partial metric invariance? Maybe you did . . .

Check http://comm.eval.org/HigherLogic/System/DownloadDocumentFile.ashx?DocumentFileKey=63758fed-a490-43f2-8862-2de0217a08b8

Also check https://link.springer.com/10.1007/s10865-013-9501-1

Hi Robert,

yes you are right. Partial invariance means that MI of each kind (metric and scalar) have to hold for at lest two indicators (the "marker" whose loading is fixed to one and an additional indicator).

Best,

--Holger

Holger Steinmetz. Thanks for getting back, Holger. I have to confess (and publicly at that - to my embarrassment) that I don't understand the statistics behind assessing MI - only some of the general notions. You're way ahead of me concerning the stats.

I am certainly concerned when researchers happily assume that MI exists when it probably doesn't exist.

But I am also concerned that sometimes unreasonable demands concerning MI might be placed on researchers - thus preventing interesting, and probably valid, research from being conducted or casting unfair doubt on research findings.

Thank you for your suggestions and ideas. I think I will first proceed as follows, as Holger has also suggested.

First I will specify the models. For this I will probably define 5 multiple group RI-CLPM (because I am interested in the combination of one variable with 5 other variables; group in this case would be the different cohorts). Then I will examine the logitudinal MI.

This may help you :Article Testing for Measurement Invariance: Does your measure mean t...