I am doing a concurrent validity study for a new assessment tool. I'm comparing the scores for about 200 students using a validated assessment (like the SAT) with scores on this new assessment. Does anyone have suggestions for analysis? I will do correlation between the scores on these two tests, but any suggestions or references would be appreciated.
You need to see whether the chosen concurrent tool (i.e. the SAT) is measuring the same psychological constructs as your newly developed tools. Are there different subscales / constructs within your new tools? Are there any other gold standard measures that you can use? Correlation between scores of your new tool and a suitable concurrent tool is good. You may consider whether Kappa statistics would be relevant to your tools as well. http://john-uebersax.com/stat/kappa.htm
Thank you! The assessment tool uses scrambled text, so the reading component of the SAT is a very similar measure. I am familiar with Kappa for inter rater reliability - I will look into it for this as well.
You may also want to consider if there is any external criterion that you could include in your study, and correlate this with both your new assessment and the established assessment
Thanks for this idea - when you say "external criterion", what exactly do you mean? I thought that a validated test was an example of an external criterion. This is a reading based test, so I am using (with permission) parts of the SAT reading test as what I thought was an external criterion. Can you be a little more specific?
Establishing validity by correlating your test against an established test can only go so far. Your new test A is validated by correlating it against established test B which in turn was validated by correlating it against established test C and so on; at some point some original test must/should have been validated by analysis against other external real world criteria. Depending on the test, these could be performance in exams, ratings by teachers, etc etc. All of these of course have many flaws and potential pitfalls when used as criteria, and would not be the only measure of validity you would use, but they could give useful supporting evidence for he validity of your test.
- Badges
- Science topic
- Similar topics
- Engineering