Hi Leng

Although IRT is generally put forward, there are advantages and disadvantages in two theories. For example, if you have performed in a small sample, it would not be appropriate to use IRT models ( except for non-parametric IRT models). At least 500 sample sizes are suggested to use IRT in many studies, while an ideal sample size of 1000 or more is recommended. In fact, the ideal sample size is assessed by data structure, number of items and item types. More information on the ideal sample size can be found in sources related to IRT (1- DeMars, C. (2010). Item response theory. New York: Oxford University Press; 2-Hambleton, R.K, Swaminathan, H. (1985). Item response theory: Principles and applications. Springer). However, many studies show that both theories give similar results when the test length and sample size are sufficient. It can be said that the differences that occur at least are practically insignificant.

On the other hand, in almost all scale development studies, statistics based on classical test theory are given. These two theories are now treated as complementary rather than alternatives. The main decision-making process is seen in the choice of IRT models. Decision which model can be used depends on the item type, data structure, distribution pattern, model-data fit statistics . Finally, it should not be forgotten that there are alternatives such as Rash and Mokken scaling. Nevertheless, I would like to repeat that it is a widely accepted approach to give results based on classical test theory, whichever alternative theory is used.

good luck

A simple question, but no simple answer. You could add factor-analysis and Mokken scaling to the list as well.

What to use depends on the model you expect to hold and the type of data you have. Do you have a theoretical model for the scale you want? Is it supposed to be one scale or does it have different segments/scaffolds or domains? (With segments, the 'domains' are also ordered, with domains they may overlap/be parallel).

CTT would require interval data, factor-analysis, IRT and Mokken can model ordinal and binary data as well. If you have ordinal data, why dichotomize it; you would lose information. With CTT, there is no indication how to create a score. One can test for additivity, but one can not assume item-scores may be summed.

If you lack a definite theoretical framework, you might want to explore which model holds best (for your data). I have a syntax in SPSS/R that allows me to analyze data in one run using factor-analysis, IRT (Rasch, 2PL, GRM and PCM) and Mokken in one run.

Some ideas about which decisions to make can be found in ten Holt (2014) A comparison between factor analysis (dissertation, University of Groningen).

Required sample size depends on the complexity of the model (amount of parameters to be estimated). Models with more parameters need more data. It also depends on the number of items. However when the number of items is large, no IRT model will fit.

Hi Peter,

Your response is really helpful. I was considering running both CTT and IRT to see what the model would look like and I was also planning on using R - would it be possible to use the syntax you suggested?

My scale is based on a theoretical concept (a few of them) but like I mentioned, my supervisors mentioned both CTT and IRT. I actually read in the literature that the sample size would depend on the number of parameters, thank you for clarifying that, however I am having trouble understanding exactly what is meant by a parameter.

My scale currently has 35 items - how do I know if this is too large for an IRT?

Thank you for the paper - I will read and digest.

BW,

Leng

Hi Leng Song ;

The model which pick up by you is depended to your data. For example, if your data is binary you can use Rasch, 1, 2 or 3 PLM. But your data is ordinal you can't use this models. You can use graded response model or partial credit model.

The sample size depend to your model complexity and disturbution of abilities.

Take a look Demars (2010) page 34.

In addition Coaley (2009) give some information about scale construction via factor analysis, CTT, IRT and Rasch Scaling. (Page 52-60)

Good luck.

Coaley, K. (2009). An introduction to psychological assessment and psychometrics. London: Sage.

DeMars, C. (2010). Item response theory. New York: Oxford University Press.

I assume 35 items Likert-type response (4 categories?). When you have a Rasch model, only the difficulty parameters is estimated. With 35 items that will be 35 parameters. In a 2PL model it will be 70 parameters. In a multicategorical Rasch analysis (4 categories) you will have three thresholds per item, one difficulty parameter per item, that amounts to 140 parameters. You can also free the discrimination parameter with multicategorical data and have another 35 parameters to add. So you see, it will add up. Mokken scaling does not work that way, it is less strict. In a Mokken scaling, the tracelines do not have to fit a specific form. It is only asked that tracelines are monotonically increasing and tracelines from different items do not intersect. This happens easily when discrimination parameters in a 2PL/3PL-like model are 'very' different.

Hi Leng

Although IRT is generally put forward, there are advantages and disadvantages in two theories. For example, if you have performed in a small sample, it would not be appropriate to use IRT models ( except for non-parametric IRT models). At least 500 sample sizes are suggested to use IRT in many studies, while an ideal sample size of 1000 or more is recommended. In fact, the ideal sample size is assessed by data structure, number of items and item types. More information on the ideal sample size can be found in sources related to IRT (1- DeMars, C. (2010). Item response theory. New York: Oxford University Press; 2-Hambleton, R.K, Swaminathan, H. (1985). Item response theory: Principles and applications. Springer). However, many studies show that both theories give similar results when the test length and sample size are sufficient. It can be said that the differences that occur at least are practically insignificant.

On the other hand, in almost all scale development studies, statistics based on classical test theory are given. These two theories are now treated as complementary rather than alternatives. The main decision-making process is seen in the choice of IRT models. Decision which model can be used depends on the item type, data structure, distribution pattern, model-data fit statistics . Finally, it should not be forgotten that there are alternatives such as Rash and Mokken scaling. Nevertheless, I would like to repeat that it is a widely accepted approach to give results based on classical test theory, whichever alternative theory is used.

good luck

Of course, Rasch model analyses will resolve with much smaller samples than 500. You can often get a good idea of what’s going on with an N of 50, sometimes less. Adding CTT results to IRT analyses is sometimes a mere sop to critics. You can check my publication list for a useful exemplar where CTT and Rasch are used together.

I agree with all contributions above. What you should also keep in mind

using IRT: is the calibration sample the same as the measurement sample? If yes --continue (each participant will more or less automatically receive his/her score) otherwise you must make sure that you know how to assign/calculate scores.

IRT models demand unidimensionality of the data. Use FA or even better CFA to check on dimensionality. In IRT items may contribute with different weights towards the score.

You do not state the purpose of your analysis (term paper, thesis, dissertation, article), make sure you understand the method completely to avoid problems in defending the results or reply to reviewers.

The following discussions are concerning which theory, IRT or CTT (number of corrects), can correctly measure examinees' high-stake performance:

https://www.researchgate.net/post/Which_one_should_we_trust--Item_Response_Theory_IRT_or_Total_Score

IRT is an incorrect theory for high-stake scoring. Please read this paper

(IRT SCORING AND THE PRINCIPLE OF CONSISTENT ORDER):

https://arxiv.org/pdf/1805.00874.pdf

Of course, Nan Kong , many see the IRT/Rasch world differently from you.