How can we ensure that examinations are valid and reliable? Will training of the staff on writing examination questions on its own provide that evidence? Are validity and reliability measurements limited to the psychometric tests conducted on the examination questions? What are the parameters we need to assess to ensure that we have undertaken adequately such evaluation?
You're asking a lot of questions here; I'll offer a few thoughts but leaver it to others to round out the picture. Since you don't seem to be a statistical maven, I want to suggest simple-but-effective methods.
If the examination is objectively scored (i.e., there is little or no judgment required - as on multiple-choice tests or others where there is a clear "right" answer and all other answers are "wrong"), then a traditional approach is to calculate the item difficulty index and the item discrimination index. You can find instructions easily, but in brief, here's what you do:
1. Score the tests
2. Arrange them from highest to lowest.
3. Consider the higher-scoring cases (ideally, about 27% of the total) the "Upper" group and the lower-scoring cases (same #) the "Lower" group. Ignore the middle cases from here on.
4. Item difficulty index (p): Count how many people in each group got the item "right." Add them. Divide by the total number of people in the two groups. This will yield a number between 0.00 (nobody got it right) and 1.00 (everybody got it right). The higher the number, the easier the item. In general, for most purposes, medium levels of difficulty are best.
5. Item discrimination index (D): Take the number of people in the Upper group who got it right. Subtract the number in the Lower group who got it right. Divide by the number in the Upper group. This will yield a number between -1.00 and +1.00. The higher the number, the better (negatives are awful). In general, you want values above +.30.
There are more sophisticated approaches to item analysis, but this works pretty well & is even used by many professional test developers. It does not speak to the external validity of the test, only to its internal consistency.
If your tests require lots of judgment (as with essay examinations), then you want to examine inter-rater reliability. Training your scorers is very important, but not enough. You should have your judges independently score a number of examinations, then calculate the correlations across judges. Higher correlations imply higher levels of reliability (objectivity, really).
This question (and also the fairness of the test) are discussed in detail in: http://www.apa.org/science/programs/testing/standards.aspx. Worth also seeing the web pages for the big testing companies for how they address fairness and reliability (and some for validity).
I can recommend S. Embretson's work on integrated aprroach to construct validity evidence.
- Badges
- Science topic
- Similar topics
- Psychology
- Psychometrics