From my point of view, it is not the assessment that bears a risk of inequity, but the way we understand results. Even on a static continuum of standards, we need a measurement of performance at certain point in time to check for progress.

From my (very limited) experience, it's all about the reader how uses the assessment results. A student may find grades unfair and discouraging, if he or she has put in the same effort as a colleague but received a lower result because of starting from a lower level. In turn, a student may also find grades unfair and discouraging, if he or she receives the same grade as someone else who has reached a lower absolute performance because the effort is assessed and not the outcome. The same goes for teachers, who may want to find the best student in class or the student who made the most progress in the recent year. Both can be done looking at the same grades, but you have to read them differently.

While what Michael Striewe says is clearly a statement of fact. It IS all in the reading, what Michael Scott says is the point I was trying to make. Different models of assessment, or more truly, models for representing achievement in assessment, do affect students differently, and some are more conducive to reframing constructively than others. As I said in the original statement, I have worked with a model that describes standards as a fixed continuum, through which students move, and more students read their results more positively than in the more common performance scale (A - E), and more importantly, the teachers read the results more positively, encouraging them to engage with student individually since they knew more detail about what students could and could not do. A side benefit is that the results can be easily represented graphically as progress, encouraging students to believe that they are 'upwardly mobile' and consequently reinforcing resilience and self belief without loss of integrity. Interviews with students demonstrated that they knew how good or bad they were at the subject, and that they persisted in engaging because they saw themselves progressing.

Thanks to all who responded. This seems to be an area that is not discussed well in educational circles. Yes Dweck, et al shows that there is a negative link between testing in general and self efficacy, and Dewey examines the sociology in terms of power structures but no-one seems to be talking about how to deal with it. I am embedded in an education system and know that a complete change of the system is unthinkable, so what can be done about it from within the system? Can we devise/design a system that meets the needs of the system and the participants. If you are interested, I have asked another question requesting ideas for design parameters that contribute to an effective, equitable education system.

Michael mentioned portfolios as an assessment model, yet I would suggest that a word or two of caution may be necessary in this regard. Numerous studies have reported significant validity and reliability issues associated with portfolio assessment. Moreover, it is widely believed by those without assessment training that portfolio assessment involves nothing more than having students submit representative samples of their work and then sitting down and reading the material. In reality, legitimate portfolio assessment requires more than one reader and significant training to socialize to an analytic rubric. The time involved is considerable and raises serious cost-benefit questions.

James, are you aware of the system used in Queensland, Australia and administered by the Queensland Studies Authority. The system relies solely on portfolio assessment for high stakes outcomes (university admission ranking) and has been shown in the past to have reasonable validity and reliability. It is in current use and involves panels of peers examining student work samples in 2 phases, monitoring - to see how the school is complying, and verification - to see how they have gone. Schools are generally found to be compliant with standards within a reasonable range. In fact Queensland schools have been found to be consistent with standards across Australia, when different systems are compared.

Mark,

No, I am not familiar with the Queensland system. Have there been any external published studies of its efficacy? In similar programs in the States, reliability is measured solely as inter-rater reliability, which is significantly influenced by the scoring rubric and the socialization process. Just because a measurement has inter-rater reliability doesn’t mean that it is constructively valid. Two readers can agree on a given portfolio's score using a particular rubric, but if the rubric itself is not valid, then that high reliability can be misleading if people assume that high reliability equals high validity.

James, I see your point. It is many years since I was involved directly with the QSA and have no papers to point you to but will look some up out of interest.

At a systemic level the issue of validity is interesting, as the system for checking validity of student work is peer based. Teams of teachers from a range of institutions vouch for the validity of other institutions by comparing assessment portfolios with their own. This is not a standardised process, more of an ad-hoc model, however it appears to work, probably because of a type of mass action. If enough different people vouch for something, then despite their individual differences, the decision approaches some degree of validity.

I do believe however that both validity and reliability are core principles of the QSA and the research concerned both areas.

Mark,

Thanks for your response. I'm puzzled, however, by your statement that "If enough different people vouch for something, then despite their individual differences, the decision approaches some degree of validity."

Let's consider the case of US writing instruction at the secondary level. The majority of secondary assignments focus on self-expressive writing tasks, and a majority of secondary writing teachers affirm that in doing so they are helping students attain writing proficiency. The high grade point averages of these students reflect that affirmation. Yet a variety of studies over the last several years have reported that the secondary curriculum is misaligned with the undergraduate curriculum--a serious problem considering that nearly 70% of high school graduates go on to some form of college, where more than 60% nationwide test remedial in writing. So here's a case in which the nation's secondary writing teachers are vouching for the validity of their assessments, but more objective measures, such as the National Assessment of Educational Progress, demonstrate that the teachers' assessments are actually invalid.

I'm having a hard time understanding how the sheer number of people claiming validity for a particular measure translates into actual validity of the assessment.

James, thanks for coming up with this example. It brings me back to the point I tried to make above: External validity (and fairness) of an assessment depends on the reader's point of view. These properties are defined in relation to a set of criteria (criteria for validity, criteria for fairness). While a large group of people can agree on having a valid assessment, another group can have different criteria and thus disagree with validity at the same time. Misalignments between curricula are a good example of having different criteria.

I guess, the point Mark was going to make is the question on whether there are criteria for a fair and valid assessment everyone can agree on.

Michael,

Thanks for your comments. I believe that the question does have an answer. Assessment literature indicates that validity in the context of performance assessment involves matching what one is measuring to what one is teaching. In an algebra class, for example, a teacher will explain how to multiply exponents and then will give students a test that assesses their ability to multiply exponents successfully. If the teacher were instead to give students a test that assessed something that he or she had not taught, such as how to solve quadratic equations, students most likely would fail the test, not through any fault of their own but because the teacher had not linked assessment to instruction. In this case, the teacher would have engaged in invalid assessment. The teacher's perception of validity in such an instance is based on ignorance and therefore has no merit. In my previous example regarding writing, we find the same level of ignorance on a much larger scale, but the perception/belief that they assessment is valid has no merit.

Face validity is just one of several types of validity, and it is the least rigorous. Assessment also involves construct validity, content validity, internal validity, criterion validity, and predictive validity.Construct validity is arguably the most problematic issue in writing assessment. It refers to the degree to which a given assessment actually measures what it claims to measure. More specifically, construct validity refers to the extent to which an assessment measures a higher-order skill or trait, in contrast to one or more dimensions of that trait. If, for example, we use an essay writing assignment to measure students' writing ability but then assess their performance on the basis of punctuation and documentation errors or just content, our assessment would lack construct validity. Our assessment, in other words, would fail to measure overall writing ability because it focused on indivdiual dimensions of this ability.

What emerges from the assessment literature is that there isn't as much flexibility in the criteria for determining validity as you suggest.

James, thanks for your answer. I'm glad it matches my limited knowledge in that area. I agree that much research has been done in this area and so many criteria and methods for determining validity are known.

Mark's original question was targeted towards inequity by comparing assessment of performance at a point in time vs. assessment on a static continuum. Is this a question of finding a valid assessment setup for the same thing or is this a question of having different goals, where for each of them a valid assessment setup can be found? If the former is true, than Mark's original question may be a question of validity. However, if the latter is true, inequity seems to have its own set of criteria that are independent from the ones for validity. This in turn would imply that there can be two assessment setups, each one valid for its own goal and invalid for the other goal at the same time.

I would like to think that it is possible to have my cake and eat it too. Or valid for both purposes.

Inequity in assessment is frequent in education. Two common instances: It occurs when teachers test students on material that has not been taught, and it occurs when the assessment instrument lacks construct validity. Probably the most frequent instance of inequity, however, is when teachers use grades for factors unrelated to education and learning.

My primary role is as a teacher, but I am also an inveterate experimenter in the classroom. I began experimenting with changing paradigms about 14 years ago, and while limited by political circumstances now, still test my ideas in mine and some other classrooms (I am a head of science so can affect changes within this faculty). The central driver for all of my experiments has been to mitigate systemic inequity. My experience from 2000 to 2004 where I had more capacity to change the schooling paradigm for students, convinced me that one could mitigate inequity when systems were carefully designed to change both the teacher's and student's belief systems about learning. I saw a number of students who had a poor belief about their capacity in school, change their beliefs, if not to highly positive, at least to neutral or slightly positive. I could only have an effect on these students for one or two years after they arrived in secondary school, so my thesis is that if we could begin this process earlier, we could reduce the systemic inequity by at least helping students to maintain a positive belief in their ability to improve (if not achieve).

Mark,

What you described as your "experiments" appears to resemble what the extensive research on teacher expectations and student performance has reported. Stated most simply, the research shows that when teachers set high expectations and regularly express their belief that students can meet those expectations, student performance and motivation improve significantly. Kudos to you.

Thank you James.

I do not consider myself gifted as a teacher. In fact I would describe myself as mediocre in many ways. The purpose of my experiments was mainly to determine ways to make it easier of ALL teachers, not just those who are gifted, to positively impact on student mindset. It was this aim that led me to the view that the system design parameters made a significant difference to how teachers taught and ergo to student self efficacy in the context of school. Which brings me back to the original idea that different systemic structures for assessment and reporting change the way teachers interact with students and so the way students see themselves at school. Everything I have done over 14 years convinces me that changing the assessment and reporting paradigm makes a significant difference to outcomes.

Various researchers have provided reams of convincing evidence that they are indeed. Much of that work comes out of Texas (see Linda McNeil at Rice U., Angela Valenzuela at UT-Austin), where the overbearing reliance on standardized test scores (coupled with official corruption in reporting of said scores) has disenfranchised thousands if not millions of students. Just to start, testing students in English when they are not yet fluent in that language is an obvious source of inequity. Also, unequal access to the tested curriculum is another problem. I could go on and on but there are several good books on the subjects ("Standardized Minds", "Contradictions of Educational Reform", "Leaving Kids Behind", etc etc etc)

I have very definite opinions on this topic. Tests like NAEP are used to identify quartiles or quintiles and students are placed in categories. (We have 20 years of these tests all saying the same thing) The hype around NAEP type exams says you will know if your student is "career/job ready". There is absolutely no validity to this statement and the tests are misused. (See E.D. Hirsch on misuse of tests in the U.S) Politically, the tests are used to find the lowest quintile so that you know which children should go to the back of the food stamp line ; you can call them a name "failed" "flunked" "lazy" "Unmotivated" and that is used to justify that they don't even deserve food stamps. The tests are not useful for educational program design or working with a student who is having difficulty in the middle school with science, for example. Much of my opinion will be based on experience but also on my reaction to extreme political ideology so it has not place here where we need to gather evidence. I will respond through aol or google email to discuss the important political dilemma for the mayors when 4,000 special education children in Virginia are told they "failed" the test. It is happening up and down the east coast of the U.S. and it is reprehensible.

My understanding of the history of American education is that it always has been political to some degree. What we have witnessed over the last three decades is a monumental shift away from community-based education to state and federal control. My sense is that the politics of education has escalated as a result. High-stakes testing is just one factor in this change, but it is a significant one. Admittedly, children's academic success is a complex matter, involving SES, parenting style, motivation, teacher efficacy, etc., but what is too often not discussed is the issue of preparation. When classes run from 8 to 2, when there are half days for teacher meetings, and when schools devote 4 or more weeks preparing students for the mandated tests, we should not be surprised that many students finish school knowing next to nothing--or worse yet, dropping out by the end of 10th grade. Teachers, administrators, school boards, and parents have been complicit in ensuring the failure of our public schools. They eagerly gave up community control in exchange for federal dollars.

I know of no assessments (tests) that actually provide teachers anything other than a rank-ordering (or percentile ranking) of a students performance. All the testing done is schools is largely a waste of time and money from an instructional perspective. A waste because if the teacher does not know who is doing well and who is not, they should quit teaching. There are evaluation tools, such as running records data, that actually do improve teaching and student learning but such tools are literally ignored by policy makers and by too many educators. Schools are drowning in data and learning is not improving. How many more years will we continue to waste the time and money we now waste on testing?

Tests--like education itself--are indeed related to ranking students. Both are sorting mechanisms. The state-mandated testing that has saturated public education over the last 30 years or so, however, has less to do with ranking students and more to do with ranking teachers. This testing is arguably not about improving students' learning but is about determining where state and federal dollars will go as "teaching incentives." Schools and their administrators fight to get these dollars for obvious reasons, hence the very public and shameful cases over the last several years of teachers and administrators changing test answers in an effort to raise overall scores. I suspect that the situation will get worse when CCSI is fully implemented next year.

All of my work over the last I5 years has confirmed for me that testing, and reporting on testing as is currently used in most education systems is a waste of time and worse actually positions some students to opt out of learning by disengaging. I think it MUST be possible to create a system that is useful to bureaucrat without positioning students negatively. It is probably obvious from other parts of my work that I think I have at least been able to outline some characteristics of just such a system.

Interesting insights! In my book for Pearson Publisher, "Student Assessment that Works: A Practical Approach," I tackled this question within a three-pronged pathway. 1. assessment works best when approach is collaborated among all stakeholders -- and I've discovered ways it can be done successfully. 2. Authentic practices align the learning and the testing into active responses to real-world problems. 3. Learning tasks -- when well formed can double as assessment tasks (in ways that increase equity).

I've spent a career with these issues and am always so glad to learn from the discourse! Thanks Mark and all! Best, Ellen Weber

Thanks Ellen, I agree that there are ways to design testing that are more equitable and you have described some issues. I think that the way results are reported on is a s7gnificant problem as well.