Hi Gary Stobbs, It is vital for the examiner to know what s/he expects in an Essay other than the topic, it is equally important for an examinee to know what is expected of him/her to write in an argument. 

I facilitate the learners to write an introduction to the essay in an overview form, in the body of the essay, they can take the stance they like (far/against) but they have to substantiate whatever they state. After making their point of view clear, they need to conclude, in an overview form, different than the introduction, in the sense that at the point of conclusion, they need to have considered all the relevant point within the time frame and the word limit.

Other than the Form and Content of the essay, I have promoted the significance of meaningfulness, grammatical competence, communicative competence, intelligibility, and task completion among them. This does not mean that they give me 100 % results, but they do try to consider these descriptors to get their thoughts across in a reasonable way. They are non natives, and they keep on enhancing their linguistic competence, whenever context and time allows them. Facilitators need to accommodate their achievement on scales, and encourage them. Assessment can never be ignored. The highlighted features can be tried as assessment scales, practice leads to fair marking.

Hi, Gary. The magic keyword is "rubric". So Google "english essay rubric" and add any qualifiers that relate to your specific needs such as IELTS or ESP.

Thanks that was quite interesting to me too.

Regards

Carmen Camilleri

In each category of the rubric, the characteristics meriting various point levels should be described carefully to promote consistency among evaluators.

Your question is interesting. We all face challenges in assessment ranging from purpose and consistency to applicability.

The following short links may help as you encounter different perspectives at your university. Some of the quotes are real gems. Alex Nottingham (2013) has a cogent piece. A favorite assessment quotes in the first link is about assessment: “...a continuously improved means to carelessly examined ends...” (Merton 1964 p vi in Madaus, 1993) The second link addresses some of your concerns directly. You might note what Tevis and Popham observe: "Perhaps the most serious problem with alternative assessments in connection with grading is the difficulty in maintaining consistency among different students (Popham 1995). Realistically, though, the process of grading, with its inherent subjectivity and arbitrariness, is perhaps the major problem itself." Tevis continues: "Assessment conducted merely for accountability reasons is not instructionally sound. To be meaningful, the act of assessment must in some way enhance the learning process...” (Tevis 1996)

Some of the experiences that changed my perspective on assessment were participating (i.e., as both a presenter and learner) in Higher Learning Commission (HLC) workshops and meetings on assessment. To condense a long story, we have used external professionals in addition to English professors to evaluate the work of our economics students at ESU. The external professionals use their organization's standards in evaluating and providing feedback on the student's work. Significant differences exist between English professors and high-level executives. Our students quickly learn that it is challenging to satisfy both. They also seem to learn much faster when they realize who is reading their work. This is one example of how assessment can accelerate the learning process.

Although it may not as easily apply in your EAP program now as it does in our economics program, I can envision ways it might. The challenge is that it takes considerable time an effort on the part of the evaluators along with students being interested in the assessment as part of the learning process. I know this is more than you asked; however, your question is interesting and sparked an interest.

https://teachingeap.wordpress.com/2013/04/25/assessment-in-eap-a-continuously-improved-means-to-carelessly-examined-ends/

http://www.edpsycinteractive.org/files/assess.html

I agree it´s a very interesting question.

We have had problems not with the rubric, but with it´s aplicability. We have several teams of evaluators, formed by three proffesors, and I have to admit that the same essay can obtain different marks, depending on the team that has evaluated it. I´m afraid it is a big problem, because of the subjectivity that the aplication of a rubric has (at least our rubrics).

Still no solution for the problem, soon we will have some meetings to try to resolve it.

Thank you for your answers so far

I did think about being more specific about the rubric we use but I wanted to encourage an open discussion on the topic in general. We use a rubric that is divided into two sections titled Task fulfilment and Language use. These are sub-divided into different levels. There are descriptions of what the students much achieve to reach each level. The problem is that they include quite subjective language to describe them. "Generally sufficient" and "Mainly correctly used" for example but what does generally and mainly actually mean in numbers because that is what we are trying to quantify. My favourite is "does not impede understanding". As an experienced teacher I can understand quite well even when the grammar is terrible but that is partly because I seen a lot bad grammar but also because as a native English speaker I can fill in the gaps better that an inexperienced non-native English speaker. 

I think that you have to be very legal minded to unambiguously populate such a rubric.

Further, you have to put your head on a block and not allow many alternative grades, as these can open the way to diverging decisions. Next, I believe that once you have generated a hopefully unambiguous rubric, that you will need to train markers in the use thereof, perhaps with some samples as standards. I am sure that behind the scenes, some of your students are complaining about their marks being too low. {For some strange reason students never seem to complain about their marks being too high}. Part of the trouble is that foreign language students cannot see how badly wrong their text is. For transparency, you should also present the rubric (and standard samples) to the students before they have to write the essays. You can also get them to mark each other for practice in understanding the rubric. External examiners from similar institutions can help to establish the norms.

I would like to add that there isn't much different between native English speakers and non-native speakers in EAP. Both groups have to learn how to write an academic essay. I think there are descriptors/rubics for general English. How to assessment an academic essay, I would follow the sample of established academic journals, such as HBR, Foreign Affairs and Economists, for the accuracy of the language, appropriate content and logical arguments.Perhaps, you can have a word with Dr Hui Li at the Department of Languages and Culture, XJLTU, for the local input. 

Here is an example of a rubric that contains descriptors to try to improve inter-rater reliability:  

     https://www.researchgate.net/publication/268075604_Oral_Proficiency_Rubric 

Data Oral Proficiency Rubric

Back in the 1970s, Educational Testing Service developed a protocol for writing assessment that is rarely followed today owing to shifting perspectives on assessment.  Here is a brief outline.  

The protocol is based on socialization to a norm.  The process begins when a group of evaluators meet and, as a group, agree on the elements of writing (global and local) that will become the rubric (which has a 6-point scale, with 1 being low and 6 being high).

With the rubric in place, readers are then socialized to the rubric using representative writing samples that reflect the range of rubric scores.  Upon socialization, the readers then begin looking at student papers.  Each paper is read twice, with the first score masked.  After reading, the masking is removed to reveal both scores, which must match within one point.  Any papers with deviating scores are assessed by a third reader.

The advantages of this approach are: (1) high reliability and (2) the ability to match the concept of "good writing" to the perspectives and expectations of the individual school.

In composition studies, the move away from this approach, called "holistic assessment," has been driven by several factors, but arguably the most significant is the understanding that using a single piece of writing may not provide a valid assessment of writing proficiency.

Hi Gary Stobbs, It is vital for the examiner to know what s/he expects in an Essay other than the topic, it is equally important for an examinee to know what is expected of him/her to write in an argument. 

I facilitate the learners to write an introduction to the essay in an overview form, in the body of the essay, they can take the stance they like (far/against) but they have to substantiate whatever they state. After making their point of view clear, they need to conclude, in an overview form, different than the introduction, in the sense that at the point of conclusion, they need to have considered all the relevant point within the time frame and the word limit.

Other than the Form and Content of the essay, I have promoted the significance of meaningfulness, grammatical competence, communicative competence, intelligibility, and task completion among them. This does not mean that they give me 100 % results, but they do try to consider these descriptors to get their thoughts across in a reasonable way. They are non natives, and they keep on enhancing their linguistic competence, whenever context and time allows them. Facilitators need to accommodate their achievement on scales, and encourage them. Assessment can never be ignored. The highlighted features can be tried as assessment scales, practice leads to fair marking.

Dear Gary,

Thanks for this good question.

Personally, I think There are two major problems related to your question. The problems cannot be separated except theoretically.

First, we must admit that there will be always an element of subjectivity that functions as a constant in our equation. The realistic plan is not to attempt to eliminate it which is futile by the way but rather reduce it to its least manifestation and then enable the EAP student to benefit from this gap by, for example, giving him-her some extra-marks. 

The second problem or the other face of the page if we want to speak realistically is Gramsci's notion of hegemony. Even with innocent subjectivity there will be always an element of hegemony? For instance, to what variety of  English community of speakers do I want my EAP learners to belong? Why would I teach for instance my students that double negative is wrong whereas native speakers use it almost everyday. Lastly, why do I want EAP learners to be more royal than the king?

To best understand this second part and enable Native speakers, academicians and curriculum designers-in brief the people who control discursivity in the foucauldian sense, we should keep asking/reminding them if they have acquired any native-like competency while learning a second language( Spanish, French...) though most of them are monolinguals by the way.

Only by attempting to look realistically at our faces in the mirror can we hope to reduce the effect of these two problems and hopefully balance the two sides of  our equation.

I believe EAP is mainly to prepare students to read the research of others and to write up the  research that they have done. It also is to prepare the student to listen to the lectures by their tutors and to speak in the same terminology. What I believe the assessment is trying to do is to measure how well the students will be able to do these four skills in the style and register that is expected for academic work. 

I have over the years used many different rubrics that used three different methods to calculate the final marks. I cannot say which is the best as they all have good points and each have some problems. I have taught and assessed IELTS but that is designed more for general English.

I now teach in an international university that has teachers from many different countries around the world and each are from different educational systems and may have a very different experience of those systems as every system is slowly developing due to changes in the political ideology and pedagogy of the period in which they studied.

It is often difficult for EAP teachers to know exactly what is expected of the students by the subject tutors. I am trying to prepare my students to study in the academic culture of my university but I am curious how other EAP tutors view their assessment criteria. 

Dear Garry,

I totally agree with you that ESP/EAP writing assessment must be totally different from what we generally use for rating General English writing assignments. What is certain, objective evaluation of EAP writing must be theory oriented. As an illustration, it is now stated that core words and semi-technical vocabulary items are more significant than specialized lexis.Alternatively, task type and the type of writing can influence the outcome. I think forming a scoring grid addressing the most important factors influencing the outcome as well as the examiners' professional discretion and judgments can be a possible solution. Examiners' accountability and issue of reliability should be the guiding forces justifying the selection of one approach rather than another.

Best regards,

R. Biria

As a slightly tangential remark, I have published a brief guide entitled Essay writing in 10 keywords which you can obtain from my page.

If I was assessing essay writing ability I would look to see how well a student had grasped and implemented these key concepts.

Decision for writing an essay depends on his method of thinking his contribution in his area of study & last but not the least he should be a voracious readers for the subject of his liking .

In the line of above practice ,quite often certain inspiration start in his mind to contribute something for what he has achieved in his life .In quite good cases for the said practice,in the mind the power force of intuition may surface in his mind which help him to carry out his contribution as a writer.

very interesting topic~~