Hi,

You may need to revisit the weightages for each response e.g. in first case for 1 weightage is , in a similar manner for case two , you need to give the 1 weightage to 2 and then compare the result. Note that you are doing the study at different time span hence results needs to be looked accordingly.

Regards,

Uday Bhale

Interesting question!

Here's my two cents:

1. There are several assumptions related to comparing the means of these scales, that fall into the ballpark of "a person answering scale A would have answered exactly the same if I had given her scale B". So without a rigorous RCT, it is quite tricky to empirically support this claim. Because if I understand you correctly, you have one time point where people answered to scale A, and a second where people answered to scale B, but you would actually have to use both scales at the same time point to make sure that the differences that you find are due to the scale and not the time point.

2. You could argue that this assumption may be more broadly phrased as "data gathered with method A would have led to the same findings if I had used method B instead", and look for literature on mode effects in sociological research, but you've probably thought about that already.

3. In my field item response theory is very commonly used to "link" items, groups, and time points, but you would have to check whether you can apply this methodology to your problem (e.g., google sth. like irt linking methods). This requires a bit more detailled info about your data.

However, from my reading of the literature, people often just assume that measures behave similarly over time or across groups. There is a recent systematic review in which the authors found that measurement equivalence is rarely checked. Note that your problem is even trickier because of the (apologies for the evil c-word) confounding of time and scale.

To compare datasets from different years with different Likert scales, you can rescale one of the datasets to make them comparable. You have two options:

Once you've rescaled one of the datasets, you can calculate the means and compare them directly. Keep in mind that this rescaling process may result in a loss of granularity in the data, so interpret the results with caution and consider additional analysis to assess the impact of rescaling on your findings.

... that solves the technical part of the problem :-)

However, I would argue that by rescaling the 5-pt, you'll make the implicit assumption that a person scoring a "1" on the original scale would score a "2" if she had been given the 10-pt scale instead. The assumption is tricky both conceptually and statistically, because you're missing information on categories 1, 3, ..., 9 of the scale, as Dinesh nicely pointed out. I'm sure there are good reasons why that assumption could hold, but it might be worthwile to contemplate on this for a bit ... After all there are research institutions focusing mostly on problems like these (e.g., GESIS in Germany).

Thank you guys for your answers. @Armin: I'll check out the IRT + GESIS.

The assumption was not, that they might score the same on scale A as on scale B - the point is, that the participants answered already the items on scale A (5 point Likert; time point 1). The point is, they will answer the exact same times in the future (!) on scale B (10 point Likert, upcoming time point 2).

So, the doubts I have were related to the comparability as changes in mean values MIGHT occure not only due to changes in the real life experiences but also due to new scale (I have not found the right literature that compare reliability on different scales with the same items over time or I did not use the correct search terms).

As we probably won't be able to adjust the future questionnaire to a 5 point scale for time point 2 in the future, we need to adapt our analysis.