Here are my tips:

Set the essay in such a way that the marker can easily spot whether the student has mastered the crucial elements.

Prepare a detailed marking rubric _beforehand_, but be willing to polish it if you find it inadequate when marking.

Do not read every word the students write. Look for patterns.

Example question: Write an essay on the foundations of problem-solving, starting with a French Mathematician, Philosopher and Scientist, giving your own opinions.

Model answer:

An old method, still valid to solve problems today, is the principle of "divide and conquer". It was first stated by the French Mathematician, Philosopher and Scientist, Descartes as "Divide each difficulty into as many parts as is feasible and necessary to resolve it."--René Descartes. I think that ...

Sample rubric:

1. Look for the key phrase "divide and conquer" 3 marks

2. The key name "Descartes" 1 mark

3. "Divide each difficulty into as many parts as is feasible and necessary to resolve it."

"feasible": 1 mark

"necessary": 1 mark

4. "I think that..." or "I do not think that..." followed by a logical answer: 4 marks.

Don't forget the 'uns'. Unfeasible, unnecessary, I don't think that, and so forth. I was waiting for Ian to answer here...

Thanks, Michael for your patience:

I add number 5:

5. "Infeasible" and "unnecessary" get 0 or even negative points, because they do not appear in the Descartes quote.

I polish number 4:

4. "I think that..." or "I do not think that...": either answer gets 4 marks (provided what follows is logical -- this is why machine marking is difficult!).

Ian, what is the Descartes quote about after all? I don't see it, so I don't get the idea.

@Michael: An old method, still valid to solve problems today, is the principle of "divide and conquer". It was first stated by the French Mathematician, Philosopher and Scientist, Descartes as "Divide each difficulty into as many parts as is feasible and necessary to resolve it."--René Descartes

I see now; sometimes asynchronous communication does not work properly. I was referring with my "uns" to an underlying and longer essay (according to Errol's outline of a 'short' essay) containing not just one sentence as in your example. I was implying students would think about your example a bit more and give counterexamples, for instance. Then they might use the "uns" and get some marks.

Sorry, The essay I mentally set the students was something like this: Write an essay on the foundations of problem-solving, starting with a French Mathematician, Philosopher and Scientist, giving your own opinion.

Now my rubric should make sense. I have edited it above.

OK, understood (from your first answer 20 hrs ago). Now Errol and his knights are left to mark. It all seems to come down to narrow the topic of the essay in such a way, that students don't have to many degrees of freedom to write about.

Yes, force the students to display their desired knowledge.

Open-ended essays should only be allowed in the creative classrooms! How much easier it is to mark a mindmap compared to an essay.

I like the discussion and I wish I was teaching more about the history and theory of problem solving. Because I am focussing on problem solving in computer science, I am tending to set exercises focussed on part of the process that I want them to apply. I know that although we document the problem solving process as sequential that isn't always how it is applied especially when we divide and conquer.

I am really wanting evidence that they understand the process and can apply it. The feedback that I want to give is to try and help them to develop their problem solving skills. At the moment my feedback comments tend to be individual for each student but it is taking far too much time.

I thought I had set a narrow topic but I still found students who seemed to be able to expand it beyond what was intended.

Michael, my markers and knights and princesses. :)

Thank you for being so kind to call it a discussion; my impression is, actually, most of the time Ian and I tried to find out what we were talking about ;). Be similarly merciful with those students that extend your topic and show some independence or creativity. In other threads here we lament the lack of creative thinking among them. In the end, problem solving needs both, analytical and creative thinking.

I am wondering what you could have gained from our "discussion" for your problem. Whereas I like mindmaps to spark ideas and make relationships visible, I doubt their usefulness for showing problem solving skills. But, @Errol, have you been thinking about a different way of letting your students show you their skills? For example, present them five task descriptions and ask them, which is the most appropriate, second best, etc. to solve the problem? Easy to mark but might not fit your curriculum, though.

Have you thought of peer review? Ask students to grade their own papers against set criteria and then to grade the essays of say 2 or 3 other students against the same criteria. You would only need to intervene if their was significant variance in the marking. Might also require students to comment. A facility to manage this should be available on the VLE you have, or just break the students into discrete sub groups.

@Michael: I had been thinking of doing something like that for the evaluation part of the process. Give them the requirement along with someone's interpretation and possibly a couple of solutions and ask them to evaluate the solutions. Not quite what they would do in a project situation but gets them thinking of the issues.

I might be able to do it for earlier stages since some would argue being able to read an algorithm description comes before being able to write an algorithm.

Yes, I like to see them showing creativity and connecting with ideas beyond the problem brief. I have that as a requirement to get the top grade in my marking criteria.

@Matt: I think our system supports peer review. I have never used it or peer review. Do you end up having to moderate the grades after the peer review?

@Jane: I think your suggestions are useful even with teaching assistants and fellow lecturers to help them see how you want them marked. I often find that I am doing a lot of work to moderate grades. I will have to try a practice paper approach.

Actually if being able to evaluate solutions is part of the process then peer review makes a lot of sense. I like the idea of multiple peer reviewers and taking the average. Do you include discussion between the markers as part of the process? I can see that being helpful for their learning.

@Errol, based on appealing ideas by @Matt and @Jane, I would be glad to learn whether you could manage the peer review under the time constraints you mentioned.

Errol, my ALA-Reader software may be able to 'score' this kind of essay. Do you have a grad student who I can brief on how to use the software, and then you can try a pilot comparing the software generated scores to your raters. If the correlations are acceptable, then you could routinize marking for future courses.

@Errol in the system I have experienced, the students scored the work of 5 other students, then they scored their own work at the end. If say 5 other students score each piece of work and all scores are aggregated and compared with the students own work, then you need only intervene if the scores breach a variance limit. @Roy is offering a software solution but also you might want to talk to Learning Support or Learning Technologists at Aston, who should be able to offer support. I am a bit long in the tooth now! but an additional learning outcome is 1) about peer review, and 2) about understanding and interpreting criteria and rubrics. Of course if you write and mark to the same criteria you gain a really valuable insight into how the system works to take forward to future assignments. I need to stress Errol I am not an expert, this was used on a MOOC I just completed, and I turned from sceptic to convert over three assignments. In other words it really worked for me. I also had some experience at my old university of what is called Peer Support. When it worked well it really worked.

Essays must be handed in as electronic copies to expedite fast analysis with software.

I so agree with @Matt, I also use peer review and make it 30% of the course grade. I think the feedback/scoring approach you use depends on the nature of the artifact. If it were a topic such as the flow of blood in the heart, the software would do a pretty good job of scoring the essay and pointing out deficiencies, but if you asked peers to read 5 of these essays, it would become pretty boring and may even lead to developing misconceptions if they read 5 in a row that repeat a common misconception (such as the single loop model versus the rue double loop model). Reading others work reinforces persistent misconceptions. On the other hand, if the essay prompt wants deeper consideration of the concepts, relations, and principles, the software may or may not give a good score, in those cases peer review can be especially good because of the interplay of the multiple conceptions contained in the 5 other essays, here the reviewer may learn more as @Matt reports.

Although holistic scoring is most commonly used for large-scale assessments, it also can be used in the classroom, with students serving as readers and scorers. This approach has at least two significant advantages. First, it allows teachers to reduce greatly the amount of time they spend grading papers. Second, it presents students with the rhetorical problem of writing for a real audience, not an audience of one. (For many years, I used this approach successfully to score an entire set of papers in an hour.)

The approach follows the protocol for large-scale assessment, with a few adjustments. It works best when two teachers are willing and able to work together on assessment. Students are provided an explanation of the procedure on the first day of class. Although the teachers do not have to give students the same assignments in both classes, each teacher’s individual assignments must require his or her students to write on the same topic. The reason is that when students in a given class write on different topics, validity and reliability are compromised.

When a paper is due, the teachers identify anchors for their respective assignments and produce corresponding rubrics. They then exchange papers, and the next day they socialize their respective students to the anchors and rubric for the other class. This can take up to 40 minutes in most instances. Students then start reading “live” papers. Scoring is finished very quickly, even in a class with 30 students, normally taking no more than 10–15 minutes. This is student-centered learning at its best, for it places a great deal of responsibility in the hands of students.

When I have introduced teachers to this assessment process during faculty development workshops, their immediate fear is that students will give every paper a high score. This fear is simply unfounded. When students are evaluating one another, they tend to be harsh, not generous, critics. Moreover, the teacher is monitoring students as they read and score.

Hello, This reference might be useful

Balfour, S.P., 2013. Assessing Writing in MOOCs: Automated Essay Scoring and Calibrated Peer Review™. Research and Practice Assessment, 8, 40-48. Available from: http://www.rpajournal.com/dev/wp-content/uploads/2013/05/SF4.pdf [ Accessed 11 November 2013].

Hello Professor Thompson

Considering the time and effort you have to put into providing written corrective feedback the effectiveness of which, due to the lack of interaction between the teachers and the students, is not satisfactory, I suggest that you choose randomly say six of the essays which have received low, mid and high scores and give WCF on them and let the students make use of the feedback while reviewing their own essays. In this way, you can also increase learners' motivation by promoting their autonomy. In a friendly atmosphere, group work can also lead to constructive peer feedback when learners review their essays.