It is very good question, Mark! My interest is also related in a big manner to the students' motivation in a classroom - and unfortunately, poor grades are usually the main reason why students become less involved in discussions. However, I think there is a solution.

Before I release any assignment, I always try to make as clear as possible what I am expecting to see, like "Our last quiz showed that you guys need to focus more on . I am expecting you to improve your results in this new assignment." Whenever I bring graded assignments in a class, I discuss the results of it in terms of how they have achieved it and make conclusions. Additionally, I am asking them what was unclear or difficult, and why they think they made mistakes. Furthermore, I spend a bit of time on recalling the most difficult concepts where they failed most of all and post some extra examples. In this case, when I present their results from the perspective what was achieved rather than what was failed, I observe that students react in a less stressful way. Also, when results are too bad, I usually allow them to re-do an assignment since I consider it as my personal fault (I either overestimate their capabilities or my problems were vague).

Hi Pavel,

this is a serious problem. Especially in these fields of higher education where you focus on an explicit transfer of knowledge and skills into practise, e.g. universities of applied sciences, marks do not reflect how well students will perform in their practical work. In our lessons, we are now also following a different approach. When preparing the lesson, we alredy define which knowledge and skills should be aquired and on which level. I also communicate this as learning goals in the first lesson. Further, I ask my students to monitor me, that means to give me immediate feedback if I don't deliver what they need to reach the goals. That works well as soon as students find the courage to tell me. I also made the observation if you as docent or professor have the opportunity to supervise the student over several semesters you have a quite different basis to discuss weaknesses or knowledge lacks.

I like the fact that you give your students a second chance with their assignments. At least, our job is to educate them, to support them in a professional development process and not only to tell them if they met one's own preferences. Please go on like that!

Both excellent responses in terms of managing assessment processes for maximum impact. Quality geedback, student involvement in the teaching process and multiple opportunities for success are all important stratiegies and it is good to see them being used. My question was also about the bigger picture: how can we manage attribution of success and failure at the end of the term or year, when student outcomes are reported on? In Australia, everyone knows that a C is a pass and D isn't. Is there another way other than giving them an A, B, C, D or E or 1 to 7 or is it possible to frame results so that pass or fail is not implied?

At Hanover university, a team started to develop competency-based curricula for all study programs (http://www.studiengangsentwicklung.uni-hannover.de/562.html, Site is in german)). I'm not quite sure if that implies quality feedback instead of traditional marks, but the European Bologna Reform demands on competency-based assessments. If consequently executed, marks then should be no topic anymore.

Thank you Susan,

I haven't quite got a clear picture of what they were doing yet, but it looks interesting. Competency based decisions have some similarity with my own work in this area.

I guess Susan brought an important point - the quantitative assessment of students knowledge is dictated by demands of contemporary educational system. And I feel that this system will stay for a long time (if not forever) because how else would you say who is allowed to go to a certain school, university, etc. An since we cannot do anything with it, we have to find ways how to reduce unavoidable frustration and fall of students motivation in classes due to their bad grades. This is very interesting psychological question, and I am willing to discuss it!

I disagree with your basic assertion Pavel. Unfortunately I don't have time to go into it right now, but I will later. The issue for me is one of early disengagement of students from a course of study that is useful to them beyond being a gatekeeper for the next stage. It is the difference between feeling 'not good enough to do xxxxx' and 'not good enough' in general. The difference between 'I failed to do xxx' and 'I am a failure'. More discussion would be appreciated.

The issue of grading, incl. the concepts of pass and fail, is paramount in society and has a long, long history. In Germany, for instance, the common grading scale (1 = best ....7 = worst, with 4 being threshold between pass and fail) indeed goes back to the middle ages and has the same draw-back as Mark mentioned w.r.t. the one in his country: it has become part of a culture, its interpretation is immediate (after years of exposure, starting at elementary school), and its connotation goes well beyond the actual content (a point on a quality scale).

In fact, reports like these don't tell so much about the past progress of learning as about the future consequences of random probing on some (well-chosen?) representative tasks: being allowed (pass) or not (fail) to move on in the school system or on the job. In other words, grades used in this way have got extrinsic meaning, being part of a decision process (pass or fail) with sometimes harsh consequences for the person involved. Quite natural, that the person involved will identify and feel very strongly with the implicit meaning of such grades!

Are there alternatives? Yes! As an example, look at the grading system in sports like judo, karate, taekwondo etc. They use colorful belts, starting with a white belt, symbolizing: I am still blank, just starting up my learning curve. Progress along the curve is in small steps. The teacher will only allow you to take the next exam, if you have been exercising enough and reached the required level. He/she will be watching and coaching you all the way, so s/he knows your chances well enough. Taking the exam is more of an formality, part of the graduation, not its condition. Of course, some will go faster, others will go slower. Some will proceed to the black belt and beyond, others will not reach the black belt at all, but still be exercising and enjoying their competence on their level, perhaps being allowed to train the newbies, which gives them some sort of gratification.

A similar story could be made up for music. I invite anybody who is a professional teacher of music or a practicing musician to tell his or her story. From my own experience I can only say this: whenever I watch a (grand) piano, I get an irresistable desire to sit down and start playing, even if there are people around, although I know that I'm at most a mediocre player (according to the high standards of professional players), and I'm always amazed to hear that some people liked what I improvise ...

Still, in both cases, sports and music, as we know very well, there is also a lot of competition, fair as well as unfair. Some people like it, will be motivated by it, above and beyond the immediate reward of feeling a competent sporter or musician. But what if a person reacts negatively on competition?

The point is whether grades and the like should be made public at all by the stakeholders or not, and if so, in what form and at which time. In large parts of the world, it is uncommon or forbidden to disclose bank accounts of private persons, and most people feel very reluctant to speak about their salaries and other income, while everybody accepts that it is extremely impolite to ask for such private things.

Why not adopting a similar cultural ban on grades? Why not establishing grading banks for all? Open a grade account, your grade account gets all the grades you earn, it will only be accessible and visible for you, etc. Nobody is entitled to consult it other than you, and nobody dares ask you. Perhaps this could take a bit of the stress currently associated with the grading phenomenon. What do you think about a grading bank?

Probably I misinterpret the question, Mark. In the light of your last post I can say the following. Giving a mark/grade for a specific assignment means to provide instructor's feedback to a particular student how well he did on that assessment (optimistic viewpoint) or how bad he did on that assessment (pessimistic viewpoint). Non-ideal grade is always a critics that should be presented in a way that student will work towards self-improvement in a particular direction. This is about every grade. If student gets D (as GPA) at the end of the school, then it probably means that he was not indeed that good in many courses. However, transcript usually provides more detailed overview of courses taken with separate grades for each of them. This is somewhat similar to me as colorful belts in Judo as it was pointed above.

There was a couple of talks on TED.com about the contemporary education system - http://www.ted.com/talks/ken_robinson_how_to_escape_education_s_death_valley.html

and

http://www.ted.com/talks/ken_robinson_says_schools_kill_creativity.html.

The speaker talks about the ineffectiveness of the existing system and offers a very interesting alternative (which sounds hard to implement though). Essentially, he claims that the approach where kids start to go to their elementary school at a certain age is wrong and deadly way for human development. In contrast, he suggests that kids should be tested for their readiness and maturity to start going to a certain school which will possibly eliminate high contrasts in grades in the class. I suggest to watch these talks and we can elaborate more about it then.

Thanks Pavel. Interesting ideas.

I enjoyed his talks (as always), and started a TED conversation from it. Join in that if you are interested.

I greatly appreciate the dialogue on grading as I have long thought about the impact of the type of grading on a student's intrinsic and extrinsic motivation to engage with the learning tasks. My work is set within physical education in the school system, and in the 90's, a curricular supplement was promoted by our province that included an evaluation system that focused on "participation with effort". Using this system, knowledge and skill competencies were assessed regularly, but not recorded as part of the overall grade; rather, they were communicated as information for the learner. The grade for the course was based on participation - with allowances made for students to make-up lost participation marks on their own time before, at lunch or after school. It's a bit complicated to explain the overall methods used; however, one intent was to promote intrinsic motivation to engage in class while using extrinsic goals as needed (e.g., encouraging students to make up lost participation marks by being active on one's own time). Interestingly, a study at the time illustrated a majority of schools surveyed used participation as a primary assessment method; the same study also indicated a majority of students in the survey enjoyed physical education (and by extension, learning in physical education). No correlational analysis was carried out, but questions remain about the impact of different assessment methods on a student's willingness to engage in the learning.

Grading based on participation? Or just marking? Again and again I notice that vocabulary regarding assessment is sloppy and confusing (cfr. see my lengthy contributions with ETS as well as International Association for Educational Assessment Network @ LinkedIN). Fortunately, many discussions are extended enough to reconstruct intention out of context.

Please note that (final) grades for a course are an instrument of (external) policy, neither for intrinsic motivation nor for formative assessment as implied by the phrase "The grade for the course was based on participation - with allowances made for students to make-up lost participation marks on their own time before, at lunch or after school". This description implies formative assessment, by definition, but it is also confusing because grades can (should) never be marks or scores - or rather the other way around.

Furthermore, IMHO, using participation (attendance?) - however qualified - as a proxy measure of competence appears far-fetched, if not dangerous. At one university I am working for, attendance lists are used, but not for assessment of learning progress, outcome, competence or any such goal. Using scores or marks (not grades!) during formative assessment may be effective for spotting students who need attention and help, but most effective feedback in terms of learning is always qualitative (what?, how?, where?), not quantitative (how much?).

Unfortunately, many students are already 'trained' to expect pseudo-quantitative grades and nothing but grades as feedback instead of qualitative content-based and objective-based information about their learning outcomes. When students come to me and ask "What is my grade?" I always counter "Sorry, grades are not my business, but if you want to know about your learning progress, please let's sit down and discuss your latest results!".

Joannie, I agree with Paul and am not a fan of using participation for grading, however participation is the most important issue. Without participation/engagement, there is less opportunity to learn, so in that sense I agree with you. And again Paul is correct that discussing grades in feedback is counterproductive. HOWEVER what I am trying to tease out is whether different paradigms for reporting allow us to do both simultaneously. Specifically the fundamental difference between progress grading, vs achievement grading. As yet I can find almost no research into the former paradigm and yet I believe it can have a profound impact on student engagement, without (much) loss of amenity for educational administrators. I have just created a new topic called Educational reporting to distinguish it from assessment if people want to use that as a topic for discussion.

@Joannie and Mark : I agree with both of you, that there is a place - indeed an important place - for progress reporting in addition to achievement reporting.

In fact, wenn I see it from a measuring perspective, the difference is almost trivial, while from a pragmatic or didatic point of view, it may indeed - as Mark suggests - be much more profound and neglected in the educational field. However, in the psychological research literature, measuring progress is a standard topic for decades, especially in the literature on child development and in developmental psychology in general. I remember from my days as a student in mathematical psychology, that we had to study the principles of modeling such developmental processes. I guess, we could learn and transfer a lot of those approaches and insights, give it the right twist and have a sound basis for progress reporting in education. But: this is just a hypothesis, someone has to take the trouble to try it out.

Why then did I say, that the difference between progress reporting on the one hand and achievement reporting on the other hand is almost trivial? Because the 'only' difference is where you place a (the) benchmark to compare previous learning state with current learning state. I assume, here and in the sequel, that you have some reliable way of observing learning states, perhaps even measuring some aspects of it on a simple scale.

---- previous state ----------------------------------------------- current state ----

---- benchmark

In progress reporting, your current benchmark is of course the previous learning state, which is the last one you reported about. Thus, your benchmark is necessarily moving along as learning goes on. So what you are asking is, whether the difference between current learning state and previous learning state is positive. If yes, the student is indeed progressing, otherwise he or she is falling back, and probably needs remedial teaching. This approach fits naturally in the pretest-posttest paradigm of any intervention scheme. The same intervention (=teaching) can thus have quite different results for different students, but still lead to a positive report. Whether this is motivating or not, depends upon the type of learner: satisfycers will see no reason to increase efforts ('all is well'), punishment-avoiders may seek to reduce efforts ('least effort for minimal gratification'), only reward-seekers will be motivated to explore their limits further ('I can even do better, if I put in more effort').

[to be continued on next posting]

[continued from previous posting]

---- current state -------------------------------------------------- future state ----

------------------------------------------------------------------------ benchmark ----

This contrasts with achievement reporting, because there we have a fixed to-be-achieved learning state as a benchmark, i.e. a goal. What we do is comparing the current learning state with this fixed benchmark. By definition then, the difference between benchmark and current learning state is almost always negative, because as soon the student has reached the benchmark, he or she has achieved the goal, and learning comes to a hold (there's no goal anymore). This clearly explains why achievement testing and reporting has this 'negative' feeling attached to it. It doesn't care from where you started from, you just have to 'achieve' the preset goal which is the same for all. Again, how a student behaves and is motivated by this approach, depends a lot on his mentality and learner type.

Thus, after all, Mark is right, when he says, that the educational consequences of adopting the one or the other 'paradigm' can be profound, but can not be predicted without taking the learner characteristics into account. This will always be a balancing act, requiring careful probing, especially in the beginning, when the teacher just doesn't know enough about his students.

Blending of both approaches?

---- previous state ------------- current state--------- future state ------------

---- lower benchmark ------------------------------------upper benchmark ----

Both approaches can be combined into a differentiated reporting policy in which the current learning state (somehow assessed) is compared both with the lower benchmark (previous state) and with the upper benchmark (future state). The result could be a mixed message of the type: 'you have shown definite progress since the last time we made a check-up (evidence...), but we have to talk about some (which?) areas in which you still have some (which?) difficulties, looking at the results of our last assignment/test (which results?)'.

Whether such an approach is feasible or not, depends again upon the educational setting at large: as this is an highly student-centric sort of teaching, more like individual coaching, it will be extremely resource-intensive. Ask yourself: what is my budget, especially my time budget? This approach, we should be well aware and frank about that, is the opposite from the mainstream we see currently in technology-centric and business-centric developments like MOOC or even in such popular approaches as Eric Mazur's peer-learning lectures with a minimum of technology.

So in the end we are back to the root question of all, I think: should teaching be conceived of as a highly personal one-to-one relationship between teacher and student, or is this a basically out-dated educational philosophy, which should be replaced by a large-scale network model in which teachers' role is to provide learning opportunities of any kind (supply of e.g. courses, seminars, exercises, ...) and students are viewed as consumers who are clever enough to look for, request and use all those selected learning opportunities in their own way being solely responsible for their own learning?

I wish I had more time to thoughtfully engage in the discussion! One thing I'd like to add: there is a Native American philosophy that "lack of competence is not a bad thing - it is an opportunity to learn …" (from Brendtro, van Bockern & Martinleg's Circle of Courage writings). I really like this philosophy … difficult as it is to promote within our western education systems of learning. In the system I mentioned above, to clarify, all types of formative and summative assessments were undertaken, with qualitative and quantitative feedback provided, so students could assess their progress. Their term "grade" however, was a value associated with their percentage score - the percentage spoke only to how engaged they were in class (i.e., participating on the task each day). Thus, a "grade" on a report card spoke only to their observed level of engagement in class … how they were performing on other methods was contained in a student portfolio to be shared with parents, admin, etc, as needed. I haven't thought much about that grading system for some time and do hope to revisit the dialogue here when I can … re: the comment: "students are viewed as consumers who are clever enough to look for, request and use all those selected learning opportunities in their own way being solely responsible for their own learning" … I think that is a nice endpoint to strive for, with teacher facilitation - though I prefer the word 'learner' to 'consumer' ...

@Pavel: Those are absolutely fantastic video clips and speaks various truths of contemporary assessment strategies.

Hi Guys

Let me tell you about the assessment regime I developed teaching mathematics to the first year engineering undergraduates at a University of widening participation. At the end I'll tel you why I thought it was successful.

1. During lecture I practiced Socratic dialogue, so students had a quick informal feedback on their progress every week

2. The students were asked to use logbooks where they put in their tutorial work homework and revision notes.

3. At the end of the 1st Semester they had a written test and were asked to submit their logbooks.

4. In the beginning of 2nd Semester, during 1 week tutorial the solution to test problems was discussed and test papers were returned to students. They were also given a list of common mistakes.

5. They were asked to do a "postmortem" on their test paper, putting against each of their written answers the numbers of common mistakes they made.

6. I then assessed their postmortems and logbooks (just looking at the quantity of tutorial, home and revision work and quality of logbook itself).

7. Judging by the quality of "postmortem" the test mark could be increased by maximum 20 % . The logbook mark could be maximum 10 % of the maximum total mark.

8. If the final total mark less than 30 % students were invited in to construct a "plan for success". Students were explained that the mark should be treated only as a start of the conversation. Looking at all three documents (logbook, test paper, "postmortem") the student approach to learning was discussed. Other specific pep talk was given, depending on circumstances.

The exam results in the next semester were usually much better, sometimes spectacularly so. All these measures can be described using various pedagogical theories. My point is - here is a practical system that is not so hard to implement and is not all that demanding on teacher time. It downplays the importance of marks which, in my view and the view of many other contributors is overblown.

Larissa - i love it! And that you are teaching math ... Have you written about this elsewhere?

I like it too. What it mainly does is say 'Here you are at this point in time. How can you make it better'. And then giving them the opportunity to do so. It Encourages students to participate in their own improvement. You are still using 'marks' in the form of percents but it is the positive mindset for self improvement you encourage in students that is the most important.

What I would like to suggest is that there may be various ways such as Larissa's that a self critical teacher can create to manage this problem. How can a system be created on a State wide/system wide level, so that a positive self improvement approach can be generated in students independantly of the teacher and still meet bureaucratif demands?

Great! Maybe I overstrech the analogy, but your (Larissa's) approach reminds me of the Test Driven Development (TDD) approach in Software Engineering: Testing (or whatever you like to call it) not as an afterthought, but right from the beginning, and everybody knows it. It takes the fear/anxiety out of it, it is constructive not destructive, it induces (self-critical) thinking, and all those other higher level "skills" or "qualities" we are discussing. Great! Thanks! Who follows?

Testing is built in FOR learning, so is normalised. Mind you the importance of a test is still in the eyes of the beholder. Students will still fear tests if the results are used uncritically, or teachers verbalise issues about test results badly.

@Joannie I have a website (soundmathematics) where I share many of my experiences.

@Mark In my view, this is a society-wide problem: addiction to numbers as "objective" measures. Addiction to tests and a practically hysterical fixation on test results starts at school. It is not that hard to change. I found that a 5-15 minutes pep talk on any subject like that can open students' eyes and change their attitudes.

Except where the system itself imposes the judgement and allots a number (or letter) which is used to pigeonhole the student. It is possible for some teachers to help some students get over being judged (and found wanting) but what I am after is a way to avoid having to help them get over being judged, because they don't feel judged, merely 'measured' to help then learn and progress. Pie in the sky I know but worth it and my experience with a different system showed that the system makes a big difference.

@Mark I always tried to make it clear to my students that I did not judge THEM, just the level of formal skills they aquired. Discussing how they can start aquiring EDUCATION and by the bye improve their formal skills and therefore these numbers surely helped. Again, it is only the belief that the marks are absolute measures of an individual that is destructive.

Mark,

I found in my own education, by far the most important thing my school did was to challenge each kid at the level of development they have currently attained. Above or below and they will switch off. Since different children develop at different rates and amounts and times, development is only loosely correlated with age, and must be continuously assessed. In my time this was done by streaming and examinations in every subject every term: presumably there is a better modern alternative now.

Since assessment is required and it is unethical to collect data about students without informing them of outcomes, it is inevitable students get to compare notes.

So to your question, how to prevent the "I am better than you" attitude, and the "You failed an assessment therefore You are a failure". The answer is twofold. One goes to school culture and the second is simple process as above: If you challenge each kid at their development level, there is a life long habit of being able to overcome the odds, because you are always and only challenged at a level which you can eventually pass, It is quite hard to feel you are a failure even if you are not eventually capable of reaching an appropriate standard for University because you have a very real sense of achievement built in every year you attend school. This philosophy applies equally and successfully across the full spectrum of physical ability and "intelligence" (whatever that might be!) Some kids will reach university standard a few years early, some a few years late, some not at all, but all will have been successful at school in their own way.

The second important thing is culture (the context of learning and school values) In my time it was the recognition BY the students that the intent of streaming was not about elitism. It was the process by which each of us could be the best we could be. This was a core school value passed by the kids themselves to each new generation, and reinforced by the school head, and (by and large) the teachers. Students at my school really did internalize this philosophy, and nobody felt streaming was an I am better than you race, but a race to be your best. This is a competition of excellence within yourself, not a competition of winning or losing. It was quite common for students to collaborate and help each other out on their own recognizance. By contrast, without this culture, the very same streaming/examination method breaks down horribly in other schools into an "I am better than you nyah nyah nyah",and the method is deemed a "failure and waste of time". But wait, the method did not fail, the cultural context did!

I am a parent who has gone through "Education Queensland" recently with two children. My son I moved to a private school in grade 6 after he was continually bullied by his teacher in front of the class during grade 5. The school culture was terrible, and no redress was available without the implicit assumption that it was all my own kids fault.

My daughter I moved from public education to a private school in grade 7 because of the woeful and inadequate standards for resourcing and teaching (Science and Maths in particular) It seemed the teachers time was by and large taken up working for the administrators rather than actually teaching. She moved back to the public system in grade 10 to 12 to one of Education Queensland trial "Academies of Excellence" I was glad she finished when she did, because these academies after a fantastic start were in the process of being sabotaged by the education department.

I would be more than happy to work with Education Queensland and make myself and my children available to explain further. Education Queensland will continue to go backwards and fail to educate our children adequately until they become professional in their understanding of what education is and what it is not. No amount of extra resourcing will fix this problem. It is not about Teacher quality. It is not about the teaching method used. It is about the right process, and the right cultural values.

The outcomes can and do follow quite quickly when you get these two things right. How do I know this? I went through such a school (yes it was a public school might I add!) in the 1970's. It worked!

I can but agree with everything that eveyone has said. We seem to be agreeing furiously. So now what? How can we create something systemic that helps teachers and students see it the right way?

Mark,

It is quite simple. I am quite sure both the education department and teachers unions feel trapped by circumstance. Both know there are serious problems and kids lives are being adversely affected. But neither can get to the solution on their own from where they are, and neither trust each other enough to give ground- and rightly so. I would also be trapped if I let myself be found in that situation!

I am a process engineer by training. I understand processes that self regulate to excellence outcomes by appropriate feedback and measurement mechanisms. I would be more than willing to chair a small start-up meeting between key education stakeholders in Queensland (perhaps including the private ones) to explore this idea that an education system can also be a process that self regulates via feedback to excellence.

You may think process engineering does not have much to do with teaching and human social interactions. Wrong! The human factors in train, airline, and nuclear safety for example are vital parts of the engineered solution, and we can and do understand how to put these systems in place. It is not about how to teach - that is your job. It is rather about the systems required to progressively and inevitably drive both your people and your systems towards the best they can be - administrators, teachers and students alike!

So do you want a teaching system with the designed excellence and reliability of the airline industry, or the made it up as we go along system that causes our national road death toll? Your choice.

I have found that once people are given permission and a pathway to escape the deadly mutual embrace that creates a toxic culture, they do so with gratitude, trust and enthusiasm, and ninety percent of the work needed to change the mindset will be done willingly.

Regards,

Andrew Jonkers

Really afreshing, these comments and recommendations! I fully agree with the diagnosis. I also agree with the proposed first steps.

But action alone will not be sufficient, at least not in the long run. Dissolving the 'deadly mutual embrace' [nice metaphor BTW, reminds me of Yourdon's book "Death March: The Complete Software Developer's Guide to Surviving 'Misson Impossible' Projects] by accompanying social therapy is a further step: good to hear that we have invented social media for that!

Another part of the solution is breaking up the 'false mindsets' unfortunately strongly induced and held up by our language and fixation on words and definitions (e.g., as if learning, teaching and assessment were different THINGS just because we need now and then terms to focus in on one of those three aspects of a single coherent process).

I don't think we as human beings will be able NOT to assess, compare and measure, just by saying NO to it: it's part of our nature and culture, deeply embedded in our flesh and blood and mind.

Look at all those examples outside education: wherever you look, you will find assesment, comparison, measurement, and people love it! E.g.:

--- Sports

--- Music

--- PISA (!!!)

--- Awards

--- Nobel Prize

--- RGscore

--- ...

Why shouldn't they be able to love it when it is placed in the right perspective? As the above comments suggest, it is possible. So, YES, let's go and do it!

I am quite happy to participate in the initiative proposed by Andrew.

As identified by Mark, individuals can cope with the pass, fail paradigm but the problem is with primary schools where developmentally, children are beginning to form their sense of self efficacy. Children from poorer socio economic backgrounds who tend to have a lower opportunity to develop self efficacy vicariously or through recieved means (Bandura, 1977, 1993, 2003) get shot down in this paradigm since their only source is through experience. Well, if their experience for any reason turns out to be 'fail', they are left bereft of any chance of developing this all important quality that is crucial for academic success- no wonder they are left in vicious cycle of failure and under achievement.

I advocate a system where children (especially in poorer areas) are started off with a system where they are not judged or measured on a traditional standardised scale (they will always be playing catch-up) but rather on a personal achievement scale. They could be reckoned on their personal progress made within a given time rather than what their grade is in relation to others.

I have tried this system with knowledge of the times tables and one of the lowest achieving children in year 3 ended up becoming better than everyone (including those in year 6) at how quickly they could recall the times tables.

@ Gideon Sappor et al. : see again my comments on this all-important issue, 13 days ago. I wonder whether there is no such thing as experimental educational research (including test methodology), because the basic idea behind 'personal achievement scale' (I like this phrase!) is the well-known pretest-posttest design principle, applied to the educational context. Actually, it applies everywhere, where you 'measure' something in progress (dynamic phenomena, time variables) *) . Thanks for sharing your experience with us, Gideon!

*) aka longitudinal measurement, repeated measurement, measurement of change, etc. See the 2013 book "Intensive Longitudinal Methods: An Introduction to Diary and Experience Sampling Research" by Niall Bolger and Jean-Philippe Laurenceau ISBN 978-1462506781

http://www.amazon.com/s/ref=nb_sb_noss?url=search-alias%3Daps&field-keywords=978-1462506781

**) BTW: the converse is called: cross-sectional measurement which is used mainly for comparing units/groups/people at the same time (period), e.g. PISA

I resonate with the idea of a system with just such attributes as you describe Andrew. I recognise that the school system has generally grown on an ad-hoc basis and that it is in need of a design process to build excellence (in the system, not the students). This point by the way is the real eye opener for me. I didn't realise it but it was what I have been doing all along. Experiment on improving the system and the teachers get drawn along with it. If the outcomes improve, then the system is good. What we are actually measuring isn't the students but the system. So now we look at the desirable outcomes of the system and measure those.

I just realised as I was writing that this is what governments think they are doing, so why doesn't it work? Why don't we get the changes we all seem to agree (in principle) with? Is the problem actually sociopolitical?

Back to Andrew's key suggestion. I would love to be involved in an engineered (I would say designed) solution for the system.

And I agree with Paul that not assessing is not an option. We need to create a system that says to all stakeholders 'you are being assessed to help you improve and there is no implied judgement of you as a valuable person'. And also as Paul says the long inculcated belief systems need to be broken down. I have worked through this by the way as I described (badly, but am rewriting it) in the Bremer action research story. Changing the system is the first and most important step. The next is a long process of reframing (in a neurolinguistic programming kind of way) of stakeholders' viewpoints. This requires allowing for a free dialogue, as in Researchgate, where ideas, complaints issues are reframed with the intent of true learning. So the idea of social media (researchgate being a good example) as an important aspect of the process is a good one.

BTW most organisations have some sort of social media built in for professional discussions but they aren't used widely. Why? Fear of being judged? Probably. But also, I have heard teachers say 'just tell me what to do and I'll do it.' Note even realising that their own beluef system value added so to speak, to what they were told to do.

SO. A big job, but worth it.

PS i did start discussing the issue here (https://www.researchgate.net/post/What_are_the_key_characteristics_design_parameters_of_an_effective_equitable_education_system) if you want to move this part of the discussion (Andrew's engineering solution)

Mark,

Pretty much spot on!, although I might slightly correct a few of your points further below. You probably have some quite fixed notions about what systems engineering is, and are right now feeling more than a little nervous about "being engineered" yourselves! Don't panic. (and sorry about the long post, but I happen to think this is quite important)

In complex systems, we are not about imposed design of efficient static monolithic structure with fixed "correct" procedures and defined outcomes that must be met on pain of death (sound familiar?) - that is more or less what got you into this state in the first place! As engineers we know this does not work. It is not about treating people as cogs in a machine. Engineers are not THAT dumb (admit it!, that is what you thought we did though ;-)

Engineers treat people in their systems as, well, people! But here is the thing. We are open and transparent about it. We never ask someone to do something without a purpose in mind that they can rationalize and agree with (or not!) . If they do not respond in kind, it is usually the case that the purpose is wrong, rather than the people we asked to do it! And so we reevaluate our purpose, rather than hiring someone else more compliant to our purpose - an act that will most likely come back to bite us in the backside (remember those O-rings on the Challenger that froze up in cold weather anyone?)

a) Complex systems engineering is not a designed imposed "solution". Rather it is a designed process added to an existing system to allow its own design to evolve to agreed outcomes in a fault tolerant, stable and reasonably efficient manner - everyone within contributes to that. Design of the education system is not imposed from without , it evolves from within, and the result is a dynamic structure that can change to meet new challenges and circumstance. Things that provably work are retained, and things that do not work, or cease to work are recycled. It is very rare to throw something (or worse, someone!) out. See below for an example of why this is so.

b) We ARE measuring the students, but the students are also allowed to measure us! We are also measuring the system procedures, and the teachers and the administrators, AND we are measuring and evaluating the measurement system as well! The entire system MUST be transparent for this to work, and nothing and no-one is immune from changes required to make it, or part of it function better. It works only because of the cultural context: We are all in it together, and the purpose is not surveillance and blame - the purpose is survival for all! - to make sure that not only are we are all rowing that boat together, but that we all want to be rowing that boat together. We are clamoring to get on that boat to give it a go, because it is so much fun, because it WORKS, and because it achieves its intended function.

By "function better" I am not talking about efficiency. Efficiency is an attribute, not a goal. If you make a system too "efficient", in fact you also forgo fault tolerance. In engineering terms, an overly efficient system is actually a catastrophe waiting to happen. We would much rather have a thousand minor procedural problems - detected and fixed as we go, rather than one catastrophic failure every 10 years (as our financial systems seem bent on doing). Ever get nervous when a flight engineer comes on board to "chat" with the pilot just before departure? Don't be, the system proven to work, is at work!

The engineering of Human factors systems is all about recognizing (empathizing) and working with (not around!) the broad range of attributes and standards of quality within the work force. It is about fault tolerance and redundancy (the good kind!) - picking up small mistakes early and correcting before they are catastrophic in outcome. It is about having many ideas in order to have a great one.

So to a specific example, to make my point:

It is a quite common, albeit eminently stupid procedure to "improve" a system by chopping off the bottom of a bell curve (a particular favorite past-time of education systems!) If you did that in the airline industry we would not have enough pilots. By definition half the pilots are below average at flying an aircraft. A problem right?. NOT AT ALL, because all but 1 thousandth of a percentile of pilots are above the required standard to fly a plane safely. See how the system asked the wrong question? We asked "Are you an above average pilot?" The question we should have asked is "Can you fly a plane safely?"

Human factors engineering is all about getting the whole bell curve above an appropriate standard, and managing the remaining few percent which by DEFINITION cannot be above that standard. I say appropriate standard - no point doubling the standard and tripling the cost because of some ideology (more must be safer right?), for no extra return in actual safety. You can always raise a standard to exclude half the bell curve - this is not about improving outcomes, but about discriminating against populations.

Now just for arguments sake, lets say a significant number of pilots were in fact below standard to 'safely' fly. Do you really believe firing half the pilots is going to solve the problem?. How are the remaining pilots going to respond? They will respond by doing everything they can do to hide their deficiencies, as will the new trainees, because they have learned something! that is, being below average gets you fired.

What they have NOT learned to do is how to fly a plane safely! Now this is not the pilots fault! The system procedure to create a safe flying environment by firing bad pilots has achieved the exact opposite. The procedure itself is the problem here, and that is what system engineering attempts to monitor and prove, and....change!

An excellent description of a desirable process. I am in accord with all of those things you describe. In a sense, I have moved my views of what I did in my research from 'a system that works' to 'desirable attributes of a system that resulted in improvement'. I imagine that this system you postulate must have some variables/parameters that can be tweaked as the need arises. Is that correct? If so then there would have to be startup parameters.

Mark,

Exactly so. The difficulty comes in explaining to people that this is not imposing yet another education policy or teaching method, or one size fits all. It is rather about a method to find the right methods - it is your job (teachers administrators and students) to find those right methods, the process simply shows you how to distinguish a good method from a bad one before it causes too much harm - and this must be evidence based - repeatable and transparent!. That is the scientific method that led us from the dark ages throughout the renaissance and we abandon it only if we have a secret longing to return to those times.

You are quite correct. A process must still be implemented to assess the merits of a change compared with the merits of NOT changing. It does not happen by wishful thinking alone! It needs agreed parameters. But these parameters are universal in application (all parts of the system assessed by the same standard) and transparent. Indeed, such a rule must make immediate sense the moment you read it, both to you personally, and to the greater good of the system as well. It is the rule by which the rules are judged! simple as that. Indeed is it truly a rule worth living by! If it is not, then you need a better rule.

Not having such a process leads generally to bad choices - either no change when it is needed, or even worse, change for the worse when it is not needed! Good choices do arise by chance - just not as often as bad ones, So in general the system degrades over time, with only occasional pockets of enlightened activity.

Aha you say, surely good choices do not arise by chance! - they surely arise by clever thinking. Of course they do - but clever thinking is a chancy thing!. Your situation at Bremer is a prime example of this. Good on you for taking some research and applying it in practice to see if it works - it can be replicated. That is the scientific method! But in the greater context of schools in Queensland you would have to concede your 'win" is a chance outcome (from the point of view of a student randomly attending a school).

The big danger of clever thinking is that people are vain. They assume their clever thoughts must be true (surely it must work that way, isn't it obvious!) , So they do not need to prove it, they are so clever, and before you know it you have yet another state wide education policy causing yet another bad hair decade for teachers students and administrators alike! Even a genius can have a bad hair day and a grain of salt is the cure!

@ Andrew Jonkers : [quote] Now just for arguments sake, lets say a significant number of pilots were in fact ***below standard*** to 'safely' fly. Do you really believe firing half the pilots is going to solve the problem?. How are the remaining pilots going to respond? They will respond by doing everything they can do to hide their deficiencies, as will the new trainees, because they have learned something! that is, being ***below average*** gets you fired. [end-of-quote]

Just to make sure that I understand your point correctly: as you were talking in the beginning about "below standard", it should also be "below standard" instead of "below average" at the end of the quotation. Changing the first occurrence into "below average" would on the face of it also be consistent, but a logical impossibility, because leaving the Bell curve will just raise the average, so in the end there will still be too many "below average"!

As a trained human factors engineer in the field of HCI (Human-Computer-Interaction) I fully endorse of course your analysis and recommendations. In order to take the "engineering" concern of some of us and to stress the interaction between people (aka stakeholders) and system-in-place, one may also speak of 'human systems engineering", what again is nothing else but old friend Human Factors (in Europe better known as ergonomics).

Paul,

Not at all. I meant what I said! - this actually goes back to the original thread question if you make it to the end!

Back to the example: First predicated on the "incorrect solution"

With the wrong process in place - that is a process to improve safety by firing pilots below average, we have false statistics - it wrongly equates the standard with the average and hides this fact away so it seems as though it SHOULD work. People are fooled into believing it works because initially it does work! The first result IS a greater proportion of working pilots above standard - so quick, lets stop the monitoring (and what a clever consultant you were to have suggested all this! - by all means spread the word hallelujah) - hey it works!

But wait a little bit and it all falls apart because a complex system is not just plain cause and effect - it is cause and effect and feedback. More pilots have to be hired now to meet demand (ticket prices have skyrocketed, demand has dropped - the industry is hurting due to lack of above average pilots!) - where in heck do you get more pilots from - you can't surely imagine that the pilots who didn't have a job before the firing were any better than the ones that were! So you begin rehiring them back on contract - hoping against hope they are now all above average due presumably to the benefit of an unpaid holiday. - but don't worry, you have stopped measuring, so you now you can just 'assume' they are.

Meanwhile the pilots have been taught by the system that below average means "getting fired", so they hide their below averageness and are ashamed of it . Or even worse the above average pilots equate being above average to "safe" so they tend to push the envelope more (because hey, I am after all above average!) . Everyone hides any deficiency - and this culture is passed on to trainees. But all this only perpetuates unsafe flying - the opposite of what was intended. The reality is that nothing has changed - the standard and the average remain unchanged! It is a false cultural value perception of change that becomes endemic - an industry paradigm if you will. And planes crash as a result!, and confusion reigns, and blame is slathered around EVERYWHERE.

Now we consider the correct solution (what more or less did happen):

The correct solution is a process to monitor for below standard behavior, and step in before it becomes unsafe - this is why it is quite acceptable for a junior pilot to call a landing abort on a notionally senior experienced pilot (maybe he has developed an eyesight problem). You then provide extra training, or more appropriate cockpit ergonomics (a pair of glasses even!), and by this means the WHOLE bell curve is raised relative to the required safety standard. The average is NOT the standard! and everyone knows it, and all is good.

This actually goes to the original question. Half the pilots KNOW they are below average, but they do not see this as failure because

a) they are doing the best they can, and

b) they have met the standard required to achieve their dream job - to fly a commercial plane safely.

c) they are therefor quite comfortable to make every attempt to understand and improve their deficiencies, rather than hide them away.

Their absolute position on the bell curve is now culturally only a minor point, but one nonetheless they CAN still use to judge improvement in their striving to be the best they can be! - it is however no longer the measure of success or failure - the standard is!. See how it works, it is all a matter of the right cultural norms!

Bad policy makers continue to make a really big mistake: in real complex systems people have change imposed on them, and they in turn adjust their own circumstances - this is a feedback system and bad policy makers fail to account for this feedback which is often so big it can often swamp the initial justification for the change and indeed completely negate its stated intent and then some! (Just how much HAS that mining tax collected!)

PS the mining tax is an Australian issue where a supposedly good government policy idea (a particular tax structure on mining companies) did not meet expectations because of negative reactions from mining companies and many in the public.

Mark,

Yes the tax policy makers failed to anticipate the feedback response - whether that feedback was "justified" or not was irrelevant; a policy failure resulted.

This is what I meant that "judgement" is not actually a part of systems engineering (although measurement and comparison are!) The feedback is what it is and must be dealt with either way so there is actually no useful purpose derived from judging the "morality" of that feedback.

Presumably (not being a teacher myself) this is why in school you must be very careful about "judging" students as compared to "assessing" them - it all goes to intent, which is all about culture, which is why you have to be transparent - the students have to see that you are acting in their interest, not just be told so!

Here are some more useful examples:

Another classic example is the riots some time back in Britain - Oh but it is all these young antisocial no-hopers fault. Well yes it is. But consider this as a feedback response to policy settings, and suddenly you find society has a systems problem that needs to be addressed. Some fines and jail time, while justifiable, is not going to actually solve this problem. Unless of course you have a culture like the USA where having full jails is a good thing, because it makes sound commercial business sense. (OK I am only human and being perhaps a bit too judgmental here - I apologize) (system engineering can be used by the dark side too!)

Feedback of "true things" can also be destructive as in :"Hey I used the scientific method to work out that our problems lie in this policy setting" The bad news is you got fired for embarrassing the policy makers. Score 1 for the system, zero for you!

Andrew, the way you describe it resonates well with my views. Judgement is the problem. Assessment without judgement would be less of a problem for any student. It is true that some will infer judgement all on their own despite the best intent of the system, but that would be a lot less than the current state of affairs. Teachers maintain this state of affairs through implied judgements about quality so teacher language would also need to be examined, but in general, I think you are completely correct. Change the system and people will slowly change along with it. It matches what I describe teaching to be. Teaching is like experimenting. You do something, see what happens, make decisions about what happened, then try something else based on your observations and inferences. If the system was like that it would iterate towards the best outcomes, as long as the parameters were well determined. I suppose you also need to choose suitable moderators to make decisions about which changes were productive and which weren't, and then disseminate them. Or am I now off track?

Mark,

No I think you are right on track. Ultimately the moderation is done by everyone as a normal ongoing activity - but practically speaking it will often be delegated - however if the delegated moderator is not on board with the assessment , then they will soon hear about it.

The implementation of a suitable assessment process "for the system" is not in fact imposed by a moderator, it is first mediated (put to the system as a proposal) so that all parties learn about it and consider it. It is only after the majority accept the proposal that it is implemented, at which point selection of suitable moderators regulates itself! - the moderator is by definition transparently open to critique by the same criterion. This solves the big brother dilemma of who watches the watcher - because once everyone understands what its purpose is, it becomes culturally acceptable for everyone to watch everyone - over-enthusiastic watching will regulate itself out of existence as will lack of any watching. Appropriate is the paradigm!

Now you typically will not get 100% agreement at mediation on anything, so the question becomes what is majority agreement. I have a particular definition of this deriving from various studies of human nature, and surprisingly it is NOT what we call 50% majority - which is demonstrably undemocratic (yes you can fool most of the people most of the time) . I propose instead 70% agreement is required to enact a change. 40% agreement is require to repeal a change. In systems theory this demonstrably makes systems reasonably conservative, but still open to rapid change when required, and able to redress systemic problems quite quickly!

You can see this in polling studies - an issue becomes important only when there is more than about 60% in favor of change , and this support will only rise above 70% when a sensible policy is proposed to solve it! It sets a high standard to impose change, but a low standard to get rid of a current rule - and so the system iterates to success based on a minimal rule set. It is also stable because you cannot get flip-flop political decisions between factions. Finally this strategy is non-judgmental because it works regardless of what the topic is about. It could be about a decision to commit genocide! Wow if 70% agreed to that, then we were in trouble anyway!

There is a special case in education because the ratio of administrators to teachers to students is always by definition in the students favor (for all I know they make up more than 70% of this system all by themselves!). I am almost tempted to say so what! There is the issue of when your are of age to take an informed "adult" decision. Also in practice an equal weighting of odd numbers in factional systems promotes a good stable robust system - so maybe one vote per class? This could be practical and quick. It would also mean that the art of politics becomes a microcosm in the classroom that students can learn and experiment with in a fairly benign way if they find it of interest.

@ Andrew, Mark or anybody who feels competent to answer this question : What is the difference in standard English language between "assessment" and "judgement" in the context of educational processes?

My provisional analysis:

As a non-native speaker of English and perhaps biased by psychological jargon, I don't feel or recognize any such big difference between both terms. Assessment - as I use the word in daily life - is a weak form of measurement, weak in the sense of not always and not necessarily supported by well-developed methods and techniques, but nevertheless reliable and valid enough for practical use. It also may involve subjective elements in the sense that potentially fallible human beings are used to produce the measurement scores, instead of having an objective test device of some sort. One way or another, performing such an "assessment" always and necessarily involves a judgemental process by a human being: someone makes a statement (assessment or judgement) about a certain phenomenon, e.g. learning progress, while observing someone else performing a certain action, e.g. assignment or test. How this assessment or judgement will be received, is exclusively in the eyes of the beholder: if a student thinks this statement tells something about his personality, there is of course a mismatch between original intention and perceived intention. I don't yet see how this could tell us something about the meanings of the terms assessment and judgement per se. Also assessment can be taken to be something negative (to be avoided), and there are many occasions where judgement will be welcomed as something positive.

I think that people who are into linguistic analysis could have great fun with that question Paul. I am not sure that, even as an native English speaker, I can answer it, but It certainly makes me pause for thought, especially your suggestion that assessment is a weak form of measurement, involving subjective elements. Many in my part of the world think that it is possible to remove subjective elements from assessment and so want to avoid using internal assessments for national qualifications - only exams set externally by some sort of objective body are to count. In more informal situations, or everyday in schools, I am not sure that there is really a great deal of difference in meaning between the view of assessment you present and judgement - only that they are used on different occasions.The legal profession aside, which of course depends on what they call judgements, my initial guess (no more than that) is that the rest of us use 'judgement' in those occasions when we have to balance pros and cons, positive or negatives, strengths and weaknesses, that sort of thing, and make a decision of some kind. Teachers' ongoing assessments (certainly of formative kinds) of their students include those sorts of balances but, in my experience at least, we tend not to see them as judgements. Assessment, even if it is not really non-subjective, is often treated as if it is. We develop performance criteria, assessments rubrics and so on to make iot appear more objective than perhaps it is.

Paul,

You make a very good point. I only have my personal definition (hey everyone else does it and gets away with it!) I take assessment to be a rather benign factual observation on the basis of a measurement. I take judgement to mean a moral stance is taken on the basis of a measurement.

Example:

Assessment: Student A scored 45 on the calculus exam and has not reached the required standard to undertake the module on differential geometry.

Judgement: Student A scored 45 on the calculus exam and is a stupid lazy person who will never be capable of understanding differential geometry.

Thanx Colin and Andrew, I will be more careful in the future using the term 'judgement' while explaining the principles and procedures of my 'assessment' system PASS (see link). Even the highly mnemonic "passorfail" in the link may be associated by some people with something negative, although I hope that most people will note the irony in it, at least after reading my stuff. Question: is irony an important skill of teachers, or should it rather be avoided?

http://www.passorfail.de.vu/

Paul,

Well putting on my students hat, I would recommend teaching irony before doing it, or at least doing it while teaching it!. However I suspect if you know the difference between sarcasm (the lowest form of wit) and irony, then it is ok if teaching within a culture where humour is an accepted normal state of a teacher.

Assessment : irony

Judgement: sarcasm

Remember also your mental health and enthusiasm to turn up is important too. If it is not there, you cannot fake it. Finally, just the act of opening your mouth will cause someone to take the wrong meaning from it. Perhaps this attempt to communicate at all is the ultimate irony!

I missed out on this set of comments. A good interchange.

Paul, you are correct to question the use of terminology. Colin and Andrew, I agree with what you have said, and Andrew, I like your analogy to help clarify meaning of the terms. Paul, the reason I introduced the Researchgate topic 'Education reporting' was to help distinguish the judgement process from the assessment process.

To extend the question of terminology: Paul you are correct when you say that there is a judgement process (I would say FOLLOWING the assessment process) and that it isn't necessarily negative. I guess I was referring to judgement specifically in the negative attribution sense. Teachers (and media and othrs) judge people to be of lesser quality if they 'fail'. The other judgement to which you refer is judgement against a criterion or standard and can be negative or positive or neutral depending on the construction and attribution inherent in the criterion. In the sense of this discussion, I think it may be best to take assessment to mean the benign process of measurement, weak, strong, subjective and objective and the neutral judgements made from the measurement, where 'judgement' can be taken to be the negative attribution by people of other people as a result of that assessment. Does that work for everyone?

It appears that Andrew is trying to position stakeholders in education to stick to assessment (as i defined it) and avoid judgement by creating an engineering quality control feedback system and consequently redefing the meanings of success and failure. My definitions of these by the way are as follows: success = continued effort to improve, failure = no or inadequate effort to improve. I have been using these constructs for 10 years, and whenever I manage to sell the new ideas to students against the prevailing paradigm, I get better results. I think that these definitions would gell with Andrew's system. Yes/no?

Mark, Yes!

On reflection there is another mode to this assessment/judgement discussion - Assessment is the process. Judgement is the culture. It is when you try to mix the two you get problems.

Example:

A fundamentalist culture that implements judgmental processes will reinforce their culture but never discover the reforms needed to save that culture from destruction because they have no assessment process to do so!

A non-judgmental culture can implement an assessment process and never change in the face of overwhelming evidence it needs to to survive!

A non-judgmental culture can implement a judgmental process that guarantees equal outcomes (it is obvious we all should be right handed), but never uncovers the evidence that it might need left handed people to survive.

The only remaining alternative - a judgmental culture, and an assessment process can on the other hand gather evidence to change its judgmental standards before it kills itself in ideology. Humans seem to have this infinite capacity to believe anything, right up to the moment it kills us. Assessment is the process that saves us from our own cleverness.

Fortunately we have a head start - humanity has a long history of judging different assessment processes, and a simple correct answer exists. The best assessment processes adhere to the scientific method. What we lack experience in is applying the assessment process to the cultural judgments! Assessment of culture is a very important and often ignored part of our systems, but engineers find it costs lives if we do not consider it! (The Challenger accident being a clear example of this deficiency in action)

Thanx for clarifications on terminology, which goes even deeper or broader than I expected. We have now two visions which may be carefully combined.

(1) Judgemental processes come after assessment processes, like what we have in legal trials (at least in non-dictatorial parts of the world) or in the medical profession (anamnesis/diagnosis of symptoms versus conclusion about illness).

(2) Or assessment is a process of analysis and measurement together with their outcomes per se, while judgement has to do with a culture of distinguishing between good and bad irrespective of any rational or empirical evidence or justification for it.

In my conceptual world, (2) is equivalent to the well-known distinction between descriptive models and prescriptive (normative) models (of action, including thinking). Well-known from decision theory, or probability theory (e.g. objective probability versus subjective probability, although in the latter case - by sheer luck I guess - researchers have found out that in the end both sorts of probability should adhere to the same principles/axioms).

As to (1), it reminds me of a strange linguistic phenomenon in some (probably only indo-european) languages, that one and the same word is used for a scale on which to measure (in a non-judgemental way!) an attribute or feature of objects AND for a certain subset of values on that scale which leads to (judgemental!) exclusion of objects which have the 'false' values on that feature or attribute. Utterly confusing, if you don't recognize that.

Example: the very word 'quality'! In the question "What is the quality of this work of art?", everybody I guess will expect an answer which may lie anywhere on a scale from very bad to very good, properly defined. However, if someone tells you "This work of art has quality!" he or she probably wants to express, that he or she likes it very much, i.e. very good (at least this is implied in German or Dutch language), otherwise, it is something of a tautology. How does this 'work' in other languages?

I'll need to think on your points first Paul. The subtlety eludes me late at night. Andres, while I agree entirely, I am concerned tha humanity, with its attendant foibles mainly power and politics will get in the way of any change such as you describe. Education is one if the more highly politicised systems in this society and politicians are unlikely to let it run free unless theg see political mileate in doing so.

Mark,

Yes of course it is hard to do. Culture is a very fragile thing. But most modern politicized systems are actually just stumbling along from one crisis to the next, and they suck you into doing exactly the same thing day after day. So it is not actually that hard to just say no, and give them the mileage they want, in return for the process you need. Particularly if you are prepared to do the hard work for them and let them take the credit.

The art is to be judgmental but not confrontational - and then persistent and in their face, every morning as soon as they arrive at work! with lots and lots of other people doing exactly the same thing at them - make it a class project in social studies, can you just imagine how much fun the students would have doing real life scientific experiments on their own educational system!. It is a simple law of numbers. With one administrator vs 100 teachers and students and parents, really and truly, what are they going to do but drown in their own paperwork!

Ghandi had it exactly right with Satyagraha. All you need is the truth, the numbers, and the will, and quite frankly, if you don't have those, then you deserve to fail, because you are behaving just like the system you are trying to make better!

We should have a discussion now as to what it means to "make something better" since this goes to the heart of your question about how a lot of students seem to be spat out of our system as failures when almost all of them are in truth quite normal reasonable young people. One can somewhat argue that all the great dictators of history were all trying to "make something better" But somehow I suspect every mother tending her child's grazed knee, knows exactly what "make something better" really means! and that is the truth, and it will set you free.

Instead we seem to be stuck with this definition of better: Wow you are a FAILURE, and with this fantastic start to adulthood we have given you, don't you dare turn around and graffiti our trains you good for nothing no hoper dole bludger (such a nice normal baby 18 years ago, must be the parents fault eh!) It all feeds back on itself, but somehow it is never the systems fault!

And even if you do go to university, and create a big debt for yourself doing it , have you succeeded?. Have you passed?. Not at all. More commonly every day on my rounds of RG I see the words: "PhD - Seeking JOB"

Interesting discussion. May I add my pennies worth. Bill Gates dropped out of university. Was he a failure? And there are many others throughout history to the present. I guess what we all agree on is the need for a culture change. (If I may ask what is culture...........zzz)

But seriously, I agree with the points raised so far. I shudder to think nothing will change unless politicians leave teaching and learning to the professionals. Yes, they can come up with policies at the macro level but please leave what happens within the classroom to the experts just like a Health minister won't be interefering with a surgeon's practice in a theatre.

@Andrew Jonkers

I thoroughly enjoy reading your posts on process engineering. Three problems:

1. it is difficult to have everybody (or even most) to agree on what criteria to use to decide which educational approaches are better

2. even if such criteria were agreed upon it is very difficult to stage pure experiments and the true experiments should take decades

3. there is no alignment between personal interests of today's educational "leaders" and interests of their staff and students

So, not quite sure how employ you clever pilots example. Any ideas?

@ Gideon Sappor : "... [politicians] can come up with policies at the macro level but please leave what happens within the classroom to the experts ..." Well, isn't that what *in fact* happens?

We have this privilege of 'academic freedom', don't we? I am pretty free to do in my classes what I think appropriate, as long as I adhere to the very broadly conceived course modules (no more than a listing of topics and general goals). And yes there is this dual regime of, on the one hand, what occurs in the classroom (with its formative classroom assessment practices), and on the other hand: certain imposed formal exams and tests at the end of a course (summative assessment functions)

It is easy to circumvent the impact of, or at least downplay the role of the latter to the advantage of the former. How? Quite simply, have a deal with your students (give and take): devote a small/tiny fraction of your precious time to "teaching-to-the-test" (what's wrong with that?); that will appease all nervous students, as it will reassure them, that they will get those required points or credits; and then you require them to follow 'your course' the rest of the time.

If you can negotiate this, you will have your students on your side. I am sure, this will work: it's a win-win-situation!

Larissa,

1) If there is a real systems problem then most people will recognize it if you ask them, but none of them will do anything about it! But if you are transparent and evidence based about good tools to let them solve their systems problems they will likely try them because if the system is better, their jobs will be easier to do, and they will own their own outcome which is always a warm fuzzy huggy bear even for the most jaded department head Of course if they do not come on board in sufficient numbers there wasn't really a problem, just someone on RG thinking there was a problem.

2) Pure experiments are never required by engineers. In process engineering we are constantly trying to assess something in the face of unknowable perturbations (hey we are engineers not scientists) There are principles for doing this successfully just not perfectly (including assessing that the assessment is not good enough for purpose). Now if that part of the system was generally thought important, it needs a plan to change it so that it IS assesable. If it wasn't important then you just stop doing it, and nothing needs to be assesed.

I will let you in on a secret. In my last three years on staff at a major Australian University, I stopped looking at my in tray (yes we still had them in those days) every three months or so I took the bottom half and threw it in the rubbish bin. Never got called on it by anyone in the system in the whole three years.

Yes indeed it may take decades to get to a good solution if the problem was decades in the making but given the state of knowledge in education this is unlikely - the knowledge is there, just not implemented. Going to the solution is a controlled feedback process, most off the improvement happens quite quickly - also there is no fixed target. so the system dynamically changes to best suit new circumstance - all we are doing is providing the right tools to enable that change to happen in a robust fault tolerant reasonably efficient manner.

3) I disagree. What you see as lack of alignment of interests, I see as just general confusion resulting from lack of assessment of the system. So everyone creates their own goal and agenda. The lack of alignment is real, but it is not by design or intent, it just works out that way in the confusion, and without confusion the very same people are more than happy to work in alignment because it creates less work for the same pay scale and who wouldn't want that!

In the airline case the motivation was simple. The airlines were getting sued for ever increasing amounts of money, and the traditional method of firing below average staff, was simply starving the industry of enough people, and companies were still getting sued. So the only alternative left was to do the right thing, engineer the human factors system and the truth set them free, because the truth in aircraft is not to crash them into the ground if at all possible!

All I can say as regards education is a small patience - In Australia they now openly admit that the standard of education has fallen significantly relative to other countries If change does not happen soon by enlightened choice then I fully expect large class action lawsuits by students and parents for failure to deliver a service fit for purpose. Failing that, organize on social media, stand up, and demand change. If the problem really is there, the numbers will follow!

Gideon, It isn't enough to have freedom in the class room, though I am grateful for at least that. Freedom means that some teachers manage to do the right thing for some kids who need it, but the majority bow to the paradigm implicit in the system and since it is 'normal', never question the outcomes. The system has a profound effect, If normal in the system is pass / fail, then that is the way teachers and students see the world, with all the attendant judgements and negative psychosocial outcomes that we have been discussing. That Bill Gates managed to circumvent the system doesn't mean the system is right. There would be a million other students who contribute less to society (and to themselves) than they could because of the negative impact of poor school experiences. I have seen first hand the difficulty of changing teachers' & students' views of what they do and the ease with which they slip back to the prevailIng paradigm when the system allows them to. In 2000, the middle school science syllabus in Queensland Au allowed for a different way of constructing curriculum, assessment & reporting and the perception of pass & fail. with effort, I managed to progressively align teachers' views with what was desirable. It took a few years. Then the National Government introduced a requirement that reversed what had been done, pushing us back to the previous system. The change in teachers' & students' views & behaviours was almost immediate. They reverted to the prior set of viewpoints, attributing much more destructive meaning to pass & fail than we had been using oven the last few years. Some students became more difficult to work with again, with their attitude to learning becoming more extrinsically motivated than in the last few years. More students started 'opting out' because they knew they could not pass in the new (old) paradigm.

SO. The system is important. With a good system, more teachers find it easier to do a better job, and more students adopt desirable dispositions to learning.

From an earlier post.

Assessment is the process used to collect data about learning. (emotionally neutral, but the construct used to devise the assessment can have an effect)

Evaluation is the process of making sense of that data. (neutral, but the construct used for the criteria has an effect)

Judgement as we intend it here has a 'moral' component, and often has negative implications where the words 'should' & 'ought' are used.It is rarely emotionally neutral

Very informative thread for the education specialists...

That sounds good in theory Kenneth, however they havetheir own view of theworld that has developed after at least 8 years of a school system that reinforces perfoprmance in report cards and high stakes testing. We do everything we can to shift the emphasis towards progress, but unless there is a measurable that is commonly accepted, the students don't quite believe us. In fact, I have seen it get worse over the last few years despite all our efforts. My high level chemistry students are actually fearful of trying because of their focus on performance. They think that without an A they are nothing. There is also the confounding factor of teachers. They are themselves positioned by the system to speak in terms of performance. It takes a LOT of change for them.

The problem is with the concept of progress itself, not its measurement (cf. models of longitudinal measurement in educational psychology). I am pretty sure most people (students, parents, teachers, ...) will have difficulty to explain - for themselves or others - what is the difference between a "measurement" at a certain point in time (as on a report card) and the difference between two or more of such measurements in time, which might show progress, stagnancy or fall-back. Also because we don't have adequate means to express or visualize it (yet).

Let me give a simple example. Think about two persons talking about there whereabouts on the mobile phone (as is one of the new rituals of our day). What do they talk about? They talk about where they currently are: at a station, or between two stations. That's all. Do they talk about the velocity of the plane, train or bus? No, very unprobably, unless there is a velocity display in the train (some trains do have it here). Do they talk about accelleration of the plane etc.? No, surely not, because there is no conventional way to express that in a normal conversation.

As long as we don't have easy means of talking about and visualizing progress in education, people will stick to what they have learned to understand and value, never mind how irrelevant it may be from our perspective. We will have to invent such communication forms and idioms, and then again it will take generations before that knowledge will be as familiar as report cards ...

Actually, I found that all that was necessary was to imply progress, by changing the structure of standards. The current form of standards is comparative ie how well you do against the expected at that point in time. ie of the form: excellent, good, ok, bad, terrible. or better than average average worse than average. If however, you construct standards by describing expected performance as a continuum through which children are expected to move,as in reading inventories, then each student will be on the continuum at different places, or rather there will be clusters around the expected position with some below & some above. In practice I have had students in grade 8 (13 years old) be as much as 4 years below r in Science and Some 2 years above. The point is that at the next measurement point they will have moved to a new place, and hence perceive it as progress. The trick is to describe the standards as a big enough set of attributes to perceive progress (as learning) with a short enough time frame, The trick I figured out was to get teachers to interpolate, ie by collecting evidence of learning over a 6 month period a teacher could make a judgement that a student had progressed towards the next level, which they reported as a confidence level ie 5.6 meant 'at level 5, with a 60% confidence of reading level 6 by the next reporting point. It worked well for grades 8-10 and could reasonably be expected to work for younger grades but I am ensure of its functionality at higher grades or Uni.

What is progress? I wonder, sometimes, if there is something askew with our concept of progress. Mark is probably right that some things can be thought of as a continuum, but I am not so sure all can. Take the process of maturation, leading eventually to walking. Textbooks (at least at school level) often present this as a constant sequence – rolls over, sits up without support, crawls, pulls up and holds with aid of furniture, walks holding onto furniture, stands alone unsupported, walks. People. Also, people generally seem to think this is the norm.

Yet watch some real children. For example, some children move around commando style and stand with the aid of furniture, before they crawl. Some move around on their behinds and go through the standing sequences and walk before they are able to crawl. Paul might be onto something when he says we have difficulty explaining the difference between a point measure (your child was sitting at 6 months, say) and the difference between two occasions (your child was sitting at six months and crawling at 9 months). But then, we might have another child and the report reads, “ Your child was sitting at 6 months and then moving around on its bottom at 8 months.” Both progress, but not the same form. So it is not, as Paul says, a problem with the concept of progress but of its measurement or, perhaps more accurately in this case, its description and reporting.

Ok, maybe that is maturation and not the same as learning in educational institutions, but it does make one wonder if progress is always along a continuum – learning to write essays, learning to inquire within the norms of a discipline, learning to paint pictures, learning to debate – add your own – are these easy to put on a continuum? In fact, some of these might be graded best when the student does something unexpected or creative – that is something that can only be loosely specified- for example, “The student has shown originality and flair in this picture.” Progress is often used as if there is an endpoint or clear goal. However, unlike walking as an end point of progress in maturation, some endpoints or goals are fuzzy – we recognise success when we see it (even there, there may be some disagreement – I like the picture, you do not) but it cannot really be specified in advance. Of course, we can point out and try to guide students in imitating what people who generally write good essays or paint interesting pictures do, but that is not quite the same as setting out a continuum of progress.

Sometimes, showing the students that they are making progress may be no more than saying, “I like that….”. “I find that interesting … because….”. If you know your students, you will encourage them (find something positive), just as we do for our own children, no matter what maturational route they take. Yet again, discussion brings us back to the fact that it looks like it is the educational system’s organisation around marks or grades that introduces the feelings of failure. As a student, you are not always allowed to develop in your own way and at your own pace. Having said that, Mark’s solution above may be a good compromise in a Standards-based system.

But, I still wonder, what exactly is progress? Is it the same in all educational activities? Always a continuum, or sometimes an erratic path?

Colin, what you suggest must be right! After all Piaget's ideas were superceded by Vygotsky's. My field was science and there seemed to be a clear progression of understandings along a continuum. That being said, in practice students tended to jump forward and then consistently work at a lower level. To me that was no problem. If on average they stayed at the higher level, even if they 'jumped' a lower one, then they deserved recognition at the higher level. If not thu they didn't. I don't see why we can't choose the system that is best for the field of study, although once again aContinuum of attributes, or several continua, can still work. If a student consistently works at (say) level 6, to pluck a number from the air, but occasionally shows something at level 9, then they could be described as ' consistently working at level 6 with flashes of brilliance at level 9. ' It says what you want it to, truthfully. To the nub of your comment. I agree that progress is overrated or perhaps just plain reductionist, but in practice it seems to work at so many levels, social, psychological and educational.

I am not quite saying that progress is overrated, just not necessarily always clearly along a continuum. In fact, I think progress is vital.You are also right to suggest that sometimes things just work, even though we might have conceptual doubts about their complete validity. That in itself is a fascinating issue.

I am not sure also that Piaget's ideas have been completely replaced by Vygotsky. I still see a lot of his influence, particularly in science education. Also, his ideas did not remain static in his lifetime (he modified them himself- something not always clear in textbook representations) and his followers have continued to modify and develop them. He acknowledged the importance of social factors, just did not seem to focus so much on them himself. In fact, the 'Piaget/Vygotsky ideas' example might be one that supports the idea that progress is not always along a continuum. Piaget in vogue (60's) Vygotsky seems to replace him in minds of many (70s, 80's), thereafter, gradual revisiting of both and noting their strengths and weaknesses. I am told that Piaget and Vygotsky were keen to cooperate, but were prevented from this by the politics of the time and Vygotsky's early death, so who knows what their ideas might have looked like if this had come to pass. Progress may depend on chance also.

I find your observation that students 'jump forward' then work at a lower level interesting. One of the modifications made to Piaget by some of his followers was that people did not always function at the same level in all contexts - a difference between competence and performance. On this model, your students have achieved a new competence but do not always display it - they get by without it on occasion. This seems to me to raise issues for both practice (should we always expect our students to show their highest levels of competence?) and assessment (does it allow them/ encourage them to show their highest levels of competence)?. And if assessment does not encourage displaying full competence, does this promote feelings of failure - to return to your question?

A key point - the issue of being limited by assessment. I warn teachers of it all the time, so yes it is an issue but I think can be worked around. What seemed to be happening was that students made the leap of understanding but couldn't maintain it. Perhaps it conflicted with some other limitations in their understanding. or perhaps it was an assessment anomoly? Your other point about expectations is also of interest, I think their is an obsession with demanding high performance at all times. learning and performance or not always good bedfellows. My feeling about learning is of comfortable, gentle swelling of understanding, not a hectic performance like an athlete. Perhaps I'll ask RG. what learning feels like.

During the seven years of "working on" my assessment model (cf. several contributions scattered over ResearchGate and LinkedIn educational groups as well as publications offered here on RG) I literally went through all stages and forms of learning discussed in the previous comments.

It was all there: slow continuous progress, sudden insights thus jumps to next level of understanding (and doing), stagnation, progress here stagnation there, illusory understanding, waiting for the next breakthrough, hard working to prove a conjecture, discussing and presenting to check my own understanding, etc.

Very often, when we say "working on" we really mean "learning the hard way". There is some "continuation", but it is only in hindsight, and mostly because we downplay and forget the difficult and frustrating phases of non-progress (whatever that means again).

Excellent description Paul! Please write something in my question on what it feels like to learn.

Is there a qualitative difference between self motivated & driven learning to achieve a meaningful goal, and the most common form of learning at school, ie externally motivated & driven?

" My feeling about learning is of comfortable, gentle swelling of understanding, not a hectic performance like an athlete." Like this metaphor Mark, though metaphors tend to both illuminate and disguise features of their referent. Paul neatly describes some things it disguises. Paul's last sentence is very apt. Understanding is often argued to be irreversible - once we understand we can't 'un-understand,' as it were. However, we are probably like science textbooks - they rewrite the history of science as a march towards the current understanding, missing out all the false starts, misconceptions and blind alleys that occurred along the way.

You might find some answers to your question Mark, if you Google "Approaches to Learning".

Yes I see Colin. The articles I read seem to concentrate on students' approaches in a school like setting,but I guess I can infer that there is also a difference between these and the approach Paul (and anyone else) may take on self driven learning. Is there therefore a third (fourth/fifth.sixth) approach that is related to solving a problem for oneself?

I think this is an area in which someone more skilled (and perhaps not retired and still driven!! - applies to us also , I suppose) than myself could do something about tidying up and integrating concepts. The deep approach is generally conceived as being driven by an intention to understand. That can be, and is, applied quite widely, including a deep approach to solving problems. The reason I like this particular concept though is that it is not seen as a characteristic of the person - though thinking has moved to suggesting that we may get into the habit of using, say a surface approach, because that has been encouraged most often. However, the idea remains that the same person may take different approaches in different contexts - classrooms and their organisation, subjects (science teaching is considered more likely to encourage a surface approach than humanities), the way the learning activity has been structured by the teacher, and so on.

There are a number of other concepts around that are similar, but different (in terms of the actual concept, but also in terms of research orientation from which they derive) - Dweck's mastery and performance motivations, Zimmerman's self-directed learning, Bereiter's intentional learning (versus treating learning as an externally imposed activity or work). Piaget can also be construed as relevant - there is research to suggest that putting students in groups with different ideas as to why things behave the way they do increases disequilibrium (to use his term) in the students' minds and enhances their learning compared to those in groups who hold more compatible views. An attempt at review would, I suspect, throw up more. Reading Paul's recent article is on my 'to do' list, so I am not sure at the moment exactly how it relates to all this.

Funny you should describe science as inspiring a Surface approach, It is just that, that I have been trying to change in my courses by requiring application in context, Maybe I can say I have been Somewhat successful because I also had to reframe teachers 'conceptions of what constituted a valid response by students as it is easier to recognize rote responses than to recognise the wide range of ways that learning maybe applied. We also had to acknowledge that application in context is inherently more difficult than regurgitation and had to adjust expectations accordingly.

Interesting Mark.

Just a minor correction for the sake of clarity. It is science teaching, not science itself, that is thought more likely to encourage a surface approach. This is in part because people tend to feel that to do science people need to have a lot of concepts correctly in mind before they can apply them and this leads them towards more expository forms of teaching. Whether they really need to is an interesting question and therein lies the debate as to whether science education should use more inquiry orientated methods. Of course, there are science teachers who inspire their students and educate in ways that encourage their students away from the surface approach to the deep. Also, done with recognising the need to encourage a deep approach, I believe that expository teaching can also encourage it - think of teachers or lecturers who inspired you. From my own experience it comes down to a simple maxim - base the learning around meaningful questions and somehow get the students to take on those questions as their own. Achieve that and they will inquire, even when listening to you - they are looking for answers to what are now their questions. Easier then to encourage problem solving and other forms of inquiry also.

Of course, when I say 'simple maxim', I mean simple to state, not always simple to achieve in practice. Teaching is an immensely complex activity. Your change to requiring application in context may be one way of achieving it more often, but with the consequence of needing to change your expectations. Teaching is also a continual learning process in itself.

Your point that rote responses are easier to recognise seems also apt to me. They are not only easier to recognise, assessments based on them are easier to administer. Does that encourage teachers to base assessments on rote learning more than they should?Thinking back, we might have in my time. You and others will have a better idea of whether that seems true now. Also,I wonder (and to return to your question) if applying learning in context, simply because it can be recognised as more demanding by both teachers and students, may be less likely to induce feelings of failure. Not sure, but seems it might be worth thinking about, especially by those still immersed in practice. Does that thought match your experience?

Colin, - a couple of quick observations/ inferences. Students definitely engage better when they are asked to 'learn, so they can do a better job of applying in context, which then becomes the assessment,' In other words I don't add on assessment, The application & communication of such, becomes the end product, But, while they still seen to feel the sense of failure in response to the report card, they are more willing to try again, No data for that, just years of observation. The issue of ease of recognition driving assessment is also probably true. Paul Vossen's work could be useful here, because it looks at the general forms of decision making during the assessment process. If people understood what they were doing better it might help move towards more authentic assessment types. We are fortunate in Queensland because we have a long history of school based assessment. Not that it guarantees better decisions, but teachers are more comfortable making them, so my job is just to tweak their understanding of what they are doing, My original question is really about the issue of reporting on performance.The most common measure is of what most call 'achievement' or 'how good was the performance?' in a comparative sense. If we reported on performance more in a positional sense, ie it was 'here' on a developmental continuum. I think it could transmit the same or equivalent information to stakeholders while implying to those reported on that they were indeed progressing and more importantly could continue to do so.