Dear colleagues,
I would like to ask whether it is possible to compare the quality of different models based on the same data but containing different number of variables using IRT analysis, namely Log likelihood, AIC and BIC statistics?
Specifically, I have a model with 36 items and I am gradually eliminating some problem variables and I want to compare the overall quality of the model between each other based on the above statistics?
Is this procedure possible?
Thank you for your answer.
Yes, you can compare the quality of different models based on the same data but with varying numbers of variables using IRT analysis, specifically by examining Log Likelihood, AIC, and BIC statistics. This approach allows you to evaluate how well each model fits the data while accounting for the number of parameters. As you gradually eliminate problematic items from your model, you can calculate these statistics for each iteration. A higher Log Likelihood indicates a better fit, while lower AIC and BIC values suggest a more optimal balance between fit and complexity.
The data type (continuous/discrete/mixed), model type (linear regression/mixture/Markov chain ...), the number of available data records, possible mutual impact of respective variables (individually insignificant variables can be important in pairs/tripples ...), prior (even vague and population based) knowledge ... all this will influence quality of your results and success or failure of tools you mentioned.
If by chance your modelled IRT is continuous and the adopted model is linear-in-parameters normal (auto)regression then
@incollection{KarKul:88,
author={M. K\'{a}rn\'{y} and R. Kulhav\'{y}},
title={Structure determination of regression-type models for adaptive prediction and control},
year=1988,
booktitle={{B}ayesian Analysis of Time Series and Dynamic
Models},
editor={J.C. Spall},
publisher={Marcel Dekker},
address={N.Y. },
}
is the proper reference for you. Good news is that the corresponding software is available.
Otherwise, a specific solution coping with the inherent problem dimensionality (you should compare quality of 2^(36) possible model differing in their structure ...
In trying to use 3 different criteria for model evaluation, the important thing is to know the underlying meanings of the different criteria. Comparison of log-likelihoods is concerned only with information you have in your existing data and the accuracy of estimates and predictions available from that data, while the "information criteria" make some adjustments related to possibly obtaining extra data that would allow improved estimates and predictions to be obtained.
In practice, the different criteria can indicate different model-structures as being "best".