Hi,

It may be a good idea to run a pilot test with a small sample and check the reliability of the questions using Chronbach (SPSS).

Best Wishes!

Oral tests for language are determined by some criteria of performance or based on the sub-skills rubric so it is better to have a test and treat it like Malini said using spss package.

Dear Karwan Mostafa Saeed,

Your concern about the internal consistency of the targeted oral proficiency test is justified. On such occasions, you have to do several things. First, you have to select an already existing instrument such as IELTS speaking module. Of course, you can develop an interview of your own based on the theories underlying the issue you want to investigate. Second, you have to have a clear-cut scoring grid, which specifies what should exactly be the weight of the responses provided by the interviewees. Third, you should use two or more raters which allows you to check for interrater reliability. Finally, choose a selective number of the participants and form a panel discussion to see if the responses have been realistically provided. This allows you to do a double checking showing the internal consistency of the instrument.

Best regards,

R. Biria

I truly thank you very much for your comments on the question. I appreciate the time you took to add your views. 

Regards

I have attached an oral proficiency rubric which has been published following peer review.

Data Oral Proficiency Rubric

I am not a software expert but I must know the correct softwares to use as an academic to enhance my job. In this case, you have to record (both audio and visual) the students answer the questions orally. This is an integrated use of WeVideo and NCH softwares as is the case with Language laboratory set ups. These software programs called  "From speech to text" I think so that would help solve your problem. The computer will automatically detect, correct and give the marks. 

This is done with the IELTS exams and is now the trend internationally in language assessments.

I presume you have the individual items scores for each of your participants. It should be fairly straightforward to calculate internal consistency, e.g., Cronbach alpha, from that data using any standard statistical software package. Of course, if you have multiple raters, you might find IRT analyses more useful.

Hi, 

 In my opinion all depends on planning and instruction – the process, since the main purpose of assessment is to improve learning. Thus, planning for alternative assessment, the closest procedure to real life use of the language, is made during planning for instruction. In this sense, instruction is designed to fit the characteristics and needs of the students, directed towards a clearly defined set of intended learning outcomes. To do so, instruction should be modified according to intended goes.

I do thank you very much for adding your answers on the question. They are truly beneficial! Once again, I thank you. 

Regards, 

Hi Karwan,

Just sharing my beliefs.

Best regards,

Dulce

What is your view on that dear Dulce? Thanks. 

Hi Karwan,

I´ll be sending something more detailed soon.

Best regards,

Dulce

Dear Mustafa Saeed,

For maintaining an acceptable reliability for an oral language test, we first need to have a well-established scoring grid. Then, there is a need for several raters rather than one. In this way, we can compute the inter rater reliability. In such cases, the researcher should opt for a debriefing session allowing the raters follow a unified evaluation approach.

Best regards,

R. Biria

Dear Dr. Reza, 

I thank you very much for your valuable comment. I am guessing I am going to have more than one rater to determine the reliability of the scoring. Now, do you think it is OK if I record the respondents' speaking performance and then give the recording to more than one rater to score their performance independently? or I have to have the raters sit with the respondents during their performance? I think it might be a bit difficult to have raters/examiners rate the  respondents' performance at the spot. Therefore, I might be recording their performance and give the recording to more than one rater and only then I run interrater reliability. Do you think there is any reference to support my claim, please? 

Once again, I thank you very much. 

Kind regards, 

Karwan 

Dear Karwan, it would be of some help to consider the criteria offered in TOEFL test for evaluating speaking: Answer to question? Comprehensibility; Organization; Fluency; Grammar; Vocabulary. The top score is 5. With these data, you may reinforce reliabilty determination. Good luck! 

Dear Dr. Saeed.

Oral tests inevitably are opinion effected. So the only solution to reach a reliable score is to firsts determine an scoring criteria or grid (getting help of validated famous oral test scoring criteria like of the TOEFL or IELTS) and then use double rater scoring system that later when representing your findings you will  need to clearly discuss your inter-rater reliability to have your scoring unquestionable.

Dear Rafael Ibarra and Seyed khoyi, 

Hello, 

I thank you very much for adding your thoughts in this matter. I will surely take your comments into my consideration. Once again, I thank you! 

Regards, 

Karwan 

Brilliant ideas up there. I wonder if you could add understanding of the questions (especially 5W questions) as well. It is very hard for my Chinese students to understand the questions (ie listening), all other oral test tasks are secondary. Moreover, listening and speaking go together. So aspects of both the skills will make your test more reliable. Another thing is about including all the speaking tasks (public speaking, debate and discussion) to get full extent of the oral proficiency of the students. Finally, speaking test variables are quite reasonable. I hope it helps!

Thank you very much dear Dr. Arif for the explanation. I particularly liked the idea where you mentioned including different speaking tasks in an oral test! Thanks again. 

Dear Karwan Mustafa,

First, let me ask if you are a IT software developer.

If you are, then fine. That should help you to identify your discrimination already. 

Someone can write a very good paper in English must have the ability to comprehend by triangulation in reading writing listening and speaking. Many times, we actually assess our students on only reading and writing and not so much on listening and speaking skills. 

Go to IELTS and TOEFL sites and see the philosophy behind why we should test all four skills.

Regards!