Hello Ali,

In general, any instrument should:

1. Measure the target behavior/attribute/perception or construct as intended (and not others);

2. Yield scores that are reliable indicators of status with respect to the target of the measurement; and

3. The items/questions/stimuli clearly communicate the intended task(s) to respondents.

Numerous texts and articles are available that elaborate these points. Your query, though, appears to focus on two specific concerns associated with instrument development:

1. The dimensionality or structure of the measure (related to #1, above), and

2. How to scale the response options (related to #1 and #2, above).

These are not mutually exclusive concerns, especially when one chooses to frame the stimuli as Likert-type response options. Most would rightly consider that framework as yielding ordinal scores. IRT models do offer a way to estimate the thresholds on an interval scale, which is quite helpful.

Dimensionality may be assessed, indirectly, for IRT models, or more directly, via CFA, EFA, or multi-dimensional scaling (MDS). For Likert-type scales, SEM programs like mplus or lavaan (in R) can handle ordinal variables in the assessment of dimensionality.

Finally, Likert himself envisioned summative scores (individual item/stimulus responses added across a set of related items or stimuli) as the usual score of interest, and not so much the individual response (see this reference: https://legacy.voteview.com/pdf/Likert_1932.pdf). This implies that the set of items should be unidimensional.

Good luck with your work.

Both factor analysis (FA) and Rasch analysis (RA) are utilized to evaluate the dimensionality of a data set. They, however, have different approaches to doing so. In other words, FA tests the data fit against a specific sample, while RA evaluates it against a measurement model. FA is a correlational model and is more commonly employed to identify sources of variance. On the other hand, RA is a hierarchical implicational model that aims to measure a single construct. So, which one is better or superior, FA or RA? Well, it depends on a study’s purpose and design. That being said, several recent studies have combined the two in their instrument validation. Here are a few examples.

Leeming, P., & Harris, J. (2022). Measuring foreign language students’ self‐determination: A Rasch validation study. Language Learning.12496. https://doi.org/10.1111/lang.12496

Mendoza, N. B., Cheng, E. C. K., & Yan, Z. (2022). Assessing teachers’ collaborative lesson planning practices: Instrument development and validation using the SECI knowledge-creation model. Studies in Educational Evaluation, 73, 101139. https://doi.org/10.1016/j.stueduc.2022.101139

Testa, I., Costanzo, G., Crispino, M., Galano, S., Parlati, A., Tarallo, O., Tricò, F., & Scotti di Uccio, U. (2022). Development and validation of an instrument to measure students’ engagement and participation in science activities through factor analysis and Rasch analysis. International Journal of Science Education, 44(1), 18–47. https://doi.org/10.1080/09500693.2021.2010286

Yan, Z., & Pastore, S. (2022). Assessing teachers’ strategies in formative assessment: The teacher formative assessment practice scale. Journal of Psychoeducational Assessment, 073428292210751. https://doi.org/10.1177/07342829221075121

Good luck,

I agree with David Morse that either IRT methods or factor analysis can be used to test the uni- (or multi-)dimensionality of a set of binary or ordinal items. From my perspective, the advantage of IRT methods is that they are "full information" methods. That means that they use the entire information from the items (i.e., the observed response pattern frequencies for each response pattern) to estimate the model parameters and determine model fit. In contrast, ordinal factor analysis methods (e.g., WLSMV estimation with tetrachoric or polychoric correlations in Mplus) are "limited information" approaches because they use higher-order summary statistics (i.e., correlations) rather than the individual response pattern frequencies in the model estimation. In that sense, IRT makes better use of the full available data which is an attractive feature.

That being said, the differences are often not large in practice, and many users may find factor analysis results to be somewhat more intuitive/easier to interpret than IRT results. Model fit assessment via chi-square test of model fit also tends to be more straightforward in factor analysis. In IRT, the chi-square p values for the model fit test are often invalid (or need to be bootstrapped to be valid using parametric bootstrapping methods) due to sparse response tables. Sparse tables occur in most applications that use more than a hand full of items. See, for example

https://citeseerx.ist.psu.edu/viewdoc/download?doi=10.1.1.457.9807&rep=rep1&type=pdf

Thank you very much for your contribution. I converted the answers given to the items into five-point answers. I scored the answers hierarchically within themselves. I did EFA to discover the structural property of the measured feature. Frankly, I saw that the dimensions I predicted were supported according to the literature. I performed CFA with the same items on a different sample and the construct was acceptable. As for the results of the Rasch analysis, I applied it to the EFA data and CFA data separately. Quite interestingly, the order of importance of the items turned out to be the same. More importantly, I combined the EFA and CFA data and did a Rasch analysis, the result was again interesting, almost close to the separate data. In this way, I have combined the three analyzes. The only problem I encountered was that EFA item loads and CFA lambda values ​​were not related to the item importance order as a result of Rasch analysis.

Dear David, thank you for your reply. You have made an important contribution to the IRT approach-based review.

Dear Mohialdeen, the publications she shared were inspiring and I gave good examples. Thank you.

Dear Christian, thank you very much for the article you shared because it allowed me to interpret the ideal profile according to the error variance in my assumptions. Thank you.

Rasch is better for a unidimensional scale