I have recently taken some classes on Rasch modeling, and I understand how to conduct some of the basic techniques. However, I'd really like to see some examples of how current researchers apply the techniques in instrument validation.
These aren't so recent, but they are well written:
Myers, N. D., Wolfe, E. W., & Feltz, D. L. (2005). An Evaluation of the Psychometric Properties of the Coaching Efficacy Scale for Coaches From the United States of America. Measurement In Physical Education & Exercise Science, 9(3), 135-160. doi:10.1207/s15327841mpee0903_1
Wolfe, E. W., Viger, S. G., Jarvinen, D. W., & Linksman, J. (2007). Validation of Scores From a Measure of Teachers' Efficacy Toward Standards-Aligned Classroom Assessment. Educational & Psychological Measurement, 67(3), 460-474.
Lamb, R. (2014). https://www.academia.edu/4177232/Development of a short-form measure of Self-Efficacy in Science and Technology. Journal of Science Education and Technology. 1-17
Dear Ryan
Here are some interesting publications:
-- Bond, T.G. & Fox, C.M. ( 2007). Applying the Rasch Model. Fundamental Measurement in the Human Sciences. Lawrence Erlbaum Associates Publishers, New Jersey.
-- Liu, X. (2010). Using and developing measurement instruments in science education: A Rasch Modeling approach . Charlotte, NC: Information Age Publishing
-- Smith, E. (2004). Evidence for the reliability of measures and validity of measure interpretation: A Rasch measurement perspective. In E. Smith & R. Smith (Eds.), An introduction to Rasch measurement (pp. 93–122). Maple Grove, MN: JAM Press.
-- Rasch Measurement Transactions Contents:
http://www.rasch.org/rmt/contents.htm
For Rasch approaches to validity:
Bond, T. G. (2004). Validity and assessment: A Rasch measurement perspective. Metodologia de las Ciencias del Comportamiento, 5(2), 179–194.
Smith, E.V., Jr. (2001). Evidence for the reliability of measures and validity of measure interpretation: A Rasch measurement perspective. Journal of Applied Measurement, 2(3), 281–311.
For validity, more generally:
Messick, S. (1989). Validity. In R. L. Linn (Ed.), Educational measurement. New York: Macmillan. Messick, S. (1995). Validity of psychological assessment. American Psychologist, 50(9), 74–149.
For Rasch validity issues, search on
www.rasch.org
All you need is here, I'd reckon:
http://scholar.google.com.au/scholar?q=validity&btnG=&hl=en&as_sdt=2005&sciodt=0%2C5&cites=1659886341122930008&scipsc=1
Thanks, Trevor. I'm certainly familiar with validity in general, and I think you certainly make the essential point that validity isn't a statistic. I was more looking for specific examples of well done rasch modeling. I am aware that unfortunately there are a lot of examples of poorly designed instruments in the field, and wanted some good examples to learn from.
I appreciate all the references, everyone. Thanks so much!
Take care to separate the wheat from the chaff, Ryan. Validity starts with a theoretical argument about the nature of the variable. It argues about the appropriateness of each purported indicator and interprets the psychometric evidence in light of expectations derived from the theoretical exposition. Many take shortcuts.
As Trevor Said , Validity is an argument. This argument needs to be supported with many evidences.
In the field of vocational assessment the following article demonstrates Rasch nicely:
Athanasou, J. A. (2001). Analysis of responses to vocational interest items: A study of Australian high school students. Journal of Career Assessment, 9, 61-79. doi: 10.1177/106907270100900105
Shameless self-promotion (full text available):
https://www.researchgate.net/publication/236095657_Evaluating_Fidelity_to_the_Wraparound_Service_Model_for_Youth_Application_of_Item_Response_Theory_to_the_Wraparound_Fidelity_Index?ev=prf_pub
Article Evaluating Fidelity to the Wraparound Service Model for Yout...
PISA does NOT use a Rasch model. It is much much more complicated than that.
I agree with Trevor that validity is an argument rather than statstistical tests. We need to understand that Rasch and other IRT based models do not provide direct assessment for validity. However, these modern methods do test fundamental metrics of whether an instument forms a valid measurement scale or not. Testing the fundamental metric properties is very important first step to go on to assess the argument of validity. There are plethora of good papers on this (as you can see in the link provided by Trevor). It might be a good idea to sneak into basics on validity tests. One of the review papers I find useful is : PESUDOVS, et al. 2007. The development, assessment, and selection of questionnaires. Optom Vis Sci, 84, 663-74. This paper gives you pretty good idea about types of validity and ways to testing them.
I would recommend looking at:
- for instance the Validity presentations at the recent Institute of Objective Measurement IOMW 2014 workshop at Philadelphia: http://www.iomw.org/2014-presentations
- more generally, the Rasch Measurement Transactions website: http://www.rasch.org/rmt/, (also recommended by Trevor above).
http://www.iomw.org/2014-presentations
http://www.rasch.org/rmt/
two very recent publications - there are many to be found:
Nielsen, J. B.; Nyholm Kyvsgaard, J.; Møller Sildorf, S.; Kreiner, S. & Svensson, J. (2017). Item analysis using Rasch models confirms that the Danish versions of the DISABKIDS® chronic-generic and diabetes-specific modules are valid and reliable. Health and Quality of Life Outcomes, 15(44).
Nielsen, T.; Makransky, G.; Vang, M. L. & Dammeyer, J. (2017). How specific is specific self-efficacy? A construct validity study using Rasch measurement models. Studies in Educational Evaluation, 53, 87-97. DOI: 10.1080/07294360.2016.