Traditionally, for an item-based educational assessment, the item discrimination index has been used to represent how well individual items discriminate between test candidates of different abilities as represented by their overall performance in the assessment.
Where an assessment is not item-based, however, an educator may be interested in determining whether the distribution of total marks is looking healthier this year (in the sense of appearing to be more discriminatory) than in a previous year, when performance was heavily skewed towards high marks.
Basic methods could be employed for comparing total scores across two consecutive years, such as comparing histograms and boxplots and comparing medians, ranges and the minimum and maximum scores, all with a view to seeking evidence for improved score qualities, such as a better approximation to Normality (if that were a preferred outcome).
However, I would be interested to learn from people involved in educational assessment, including research on this topic, who have a well-defined system in place for monitoring the quality of an assessment based on analysis of total scores across a cohort, where the test is not item-based and there is an interest, nevertheless, in obtaining some evidence that the test has discriminatory power based on the standalone test scores (rather than a comparison with other scores which are alleged to represent true ability).
I appreciate that there are multiple confounders which could influence apparent differences in discriminatory power from one year to another, including factors relating to the exam candidates. However, my intention is to omit this particular discussion point for now, thanks, and start from an 'all other factors being equal' perspective.
Hello Margaret,
By non-item based, do mean holistic appraisals, or do you mean some score that is perhaps a composite of scores of multiple products (like a portfolio), or perhaps do you mean student grades, which typically reflect some ad--hoc composite of homework, quizzes, exams, and extended project(s) performance?
There are two conceptual hurdles I can see when trying to assert that some distribution (of said scores) is somehow preferable to that of another:
1. The implication that, in educational settings, outcomes ought to reflect a normal distribution; and
2. The corollary presumption that successful instruction can never yield little or no distribution in learner performance on some task. (This was the key goal of champions of the "mastery learning" movement, like Bloom, who argued that, given sufficient time and appropriate instruction, the vast majority of learners could adequately master just about any school-based skill.)
It's certainly fair game to ask the question, does score "X" differentiate among groups known to be different on some relevant attribute? As well, it's certainly fair game to ask, has the distribution of performance scores changed over time? (Educators would certainly hope that the answer to this question is more often "Yes" than "No, not so much!")
Apologies if I've misunderstood your query, but perhaps you could elaborate if that was the case.
Good luck with your work.
Thanks for your response, David. I am thinking of a final written exam paper and am focusing on the overall percentage marks. I am seeking to avoid thinking of the individual questions as items making up a construct as there are good reasons why this would not make best sense for the purpose I have in mind. I think that the idea that the distribution of the scores should look, at least approximately, Normal is a working assumption that non-statisticians adopt. I am sure that it is unrealistic. On the other hand, for the context I have in mind, I would expect level of ability and preparatory effort made by students (among other factors) to have an impact on performance across the type of cohort I have in mind. Therefore, for a fairer test, I would expect to see a shift away from skewedness in the scores (assuming other factors held constant).
I am sorry that I couldn't follow what you were saying under your item 2.
I am not seeking to discriminate between known groups, just trying to get a sense of whether the results appear (in a loose sense) to indicate that a test differentiates between students better than the same type of test in a previous year. My original query (which already includes a few basic suggestions) could be read as looking for criteria that people use in practice on a routine basis.
Hello again Margaret,
If instruction has been successful, then there should be less spread in scores, and a negative skewed distribution would be more likely than other shapes...that's a nutshell consequence of the mastery learning perspective.
On-demand writing (as in examinations) may well yield more or less spread in scores, but that, by itself may not signal a better or worse measure. Here's some simple examples as to why:
1. If a task requires skills or competence unrelated to the target skill to be assessed, then that "noise" will result in measurement error or bias, which will tend to add to score variance, but in a way that's unrelated to the measurement goals.
2. Yes, items do differ in challenge level (as do prompts for writing exams). For this reason, trying out a test before it becomes live is important, and, for high-stakes tests, vital. In other words, year-to-year drift in performance (overall and/or spread) may be an artifact of the specific measure rather than differences in ability or (true) achievement level.
I would suggest that comparison of the scores to some stable, external variable might be a better way to go than trying to read too much into the shape of the score distribution for one batch compared to that of another, when the measures are not demonstrably equated.
Good luck with your work.
I understand that the study reports qualitative data that takes into account the initial level of each student and the circumstances of the environment. I understand that the same evaluation system has always been applied. If so, the variables related to the particular abilities of each student and the environment are overridden.
Therefore, in my opinion, the secret of the success of your objectives is in your abilities to transform that qualitative information into quantitative information, since differential statistics can only be done with numbers. The rest are simple observations.
My advice is to start from the evaluation tables used in class. You must justify that there have been no changes in the evaluation criteria. And then replace each level with a number. For example, it may appear on the grid that the student is not making progress, making adequate progress, or making more progress than expected. In this case, change the levels to 1, 2, and 3. If possible, you should put 5 or 7 levels, as short scales are less suitable for differential stats.
Once you have the numbers, all you have to do is calculate the mean, the deviation, and apply the t-test or ANOVA to see if there are any differences. Also compute the skewness, kurtosis, and z-value of K-S to see if the distribution is normal or not.
Good luck with your work.
- Badges
- Science topic
- Similar topics
- Instrumentation Engineering
- Instruments