Dear Ümit, The lack of clear basic definitions is a sign of a good area in which to work!

At my university we tried to avoid retesting at all costs. It was not mentioned in the syllabus, and was not part of the student vocabulary.

The first reason was that the student may have had contact with copies of the test.

(Think: cell phone photo or under-breath voice recording!). The second reason is the added workload and the impossibility of setting an exactly equivalent test paper. The third reason is that it drags out the academic year, and the marks have to recorded in time for graduation etc.

Retesting is not the same as dealing with genuinely sick or marginal cases.

We did accept original doctor's notes (after scrutinising them for forgeries!)

We would set a new exam and get it approved by the external examiner. (External examiners are good at seeing the exam paper from the students' viewpoint and picking up errors.)

Also students who only just failed one subject, were given the opportunity to write a supplementary exam, which was probably slightly harder than the main exam, because the good questions had already been used up. Usually the supplementary results were only available early in the new academic year which was an inconvenience for the re-registering student.

(a) Re-taking the same test is pointless pedagogically because it is teaching the student to pass the particular test and not to learn. Teaching to the test is a bad practice.

I would never advocate an ungraded test as it implies that the test is unimportant, and the whole point of a test is to provide feedback to the student and to formally assess the classes progress.

Certainly, however, the lecturer or teacher must work through examples and exercises in class, giving correct answers and explanations.

(b) This (last grade) is how driver's licenses are granted. (In schools and universities, average grade is the traditional method of getting the students to put into their short term memories each part of the syllabus. The different tests each assess the next part of the syllabus. The exam is the time when long-term memories are tested.) There is no time in the crowded syllabus to stall on re-testing the same part.

(c) Sometimes (for educational research) it may be thought necessary to experiment by giving students the same test. However since learning has presumably taken place, the results must be better each time, which introduces control problems. Different but similar pre-test and post-tests can be used, provided that one controls for difficulty of the test.

I link to a reading list for you.

https://scholar.google.com/scholar?hl=en&q=test+retaking

https://scholar.google.com/scholar?hl=en&q=test+retaking

Thank you very much, Ian, for responding in such detail.

Let me be clear about why I am focusing on this issue. I started using training tests in my teaching starting about three years ago. The snapshot I attached will give you an idea how the practice works.

I am not actually sure whether this is what you mean by “ungraded test” (my training tests are graded but the grade does not count.) when you say:

• "I would never advocate an ungraded test as it implies that the test is unimportant, and the whole point of a test is to provide feedback to the student and to formally assess the classes progress."

I appreciate the concern you express. However, I believe that a student’s reaction regarding the importance of a training test is modulated by her seriousness, motivation, goal orientedness etc. (I know that I am not defining these terms, but I would in a statistical study that aimed to measure the confounding effect of such affective states). Consider the fact that the official preparatory “Triple Test” for the LSAT in the US is a universally recommended training tool. The same goes for almost all commercial SAT, MCAT, GRE etc. preparatory materials: they are chock-full of “practice tests” the grades of which do not count.

At the same time, I would be interested in seeing some serious research regarding the effectiveness of this method. The work on retesting on standardized tests like the SAT is not really relevant, because these are tests that measure existing skills and abilities (if they do). They do not focus on the effect of my kind of retesting on learning new content (maybe the optional subject-matter focused sections of the GRE would be relevant—but they might not even exist nowadays).

Why am I interested? Simply because raising the number of Training Quizzes from 1 to 2 did not have a positive effect on actual grades—in fact, the effect was mildly negative (accompanied by a drop in the number of questions addressed to the instructor). Maybe this can be explained by over-dependence on “technical tools” by today’s students. When they get a new “tool”, they feel less dependent on what the teacher has to offer…

In any case, I am trying to discover if there has been any serious statistical study of this “training” process. So far, not much. Take the first article in the link you so graciously provided: Here is what the authors say:

• The above results, of course, presume that the examinee takes the test only once. However, in a number of competency or mastery testing situations (e.g., the South Carolina High School Exit Exam), any examinee who fails the test is permitted to retake the test for a specific number of times or until he or she finally obtains a passing score.

The purpose of this paper is to study the behavior of the above errors and to provide procedures for their estimation when the examinee is permitted to retake the test that he or she fails in the first place. pp. (39-40) (Huynh, H., "Error Rates in Competency Testing When Test Retaking Is Permitted". Journal of Educational Statistics, Vol. 15, No. 1 (Spring, 1990), pp. 39-52 )

I grant that the context supports a reading according to which the issue is re-taking the exact same, identical test over and over (as philosophers would put it, we are talking about instantiations of the same test-token, not the same test-type). At the same time, I cannot believe that the State of South Carolina uses such an assessment procedure. The bottom line is, I do not think that one should be guessing the meaning of such key terms in a statistical study. The needed clear and precise definition is conspicuous for its absence.

Since I have little time to survey hundreds of articles to see if a definition is given, I gave up after about 10 articles and 2 dissertations :)

Thanks very much again. I will post separately to address some of the other issues you touch on.

Regards

Dear Ian;

Here are some aditionaI ruminations on some very interesting points you raise in your kind response:

I wholeheartedly agree when you say:

• (a) Re-taking the same test is pointless pedagogically because it is teaching the student to pass the particular test and not to learn. Teaching to the test is a bad practice.

At the same time, it is amusing (and disheartening) to observe that the academic literature, some of which is extolling the virtues of MOOCs (and hailing them as the “education of the future”) ignores the fact that this “retesting method” is commonly utilized in most MOOCs. I can personally attest to having taken 2 of them where this retesting method was in practice.

Let me also address what you say here:

• The first reason was that the student may have had contact with copies of the test.

(Think: cell phone photo or under-breath voice recording!). The second reason is the added workload and the impossibility of setting an exactly equivalent test paper. The third reason is that it drags out the academic year, and the marks have to recorded in time for graduation etc.

True, True and True :) Here are my prophylactic measures:

(1) The previous contact issue is relevant only to the actual test, and is a problem for teachers who use the same test year after year. I have 3 tests of comparable-strength for each test-subject (please see below). I switch them back and forth between being a training test and an actual test. At the same time, new questions are periodically added to the question pool.

(2) The comparability issue arises for any new test (say, created for reasons touched in (1) above.) In addition to my “nose” and the noses of my colleagues (whom I routinely consult over a cup of coffee), I try to establish “comparability" by measuring the Dale-Chall readability level of each questions in the test, and use Item Analysis of test results. Class performance over multiple semesters is compared.  I emphasize that these techniques are only useful as additional corroboration for the judgment by the team of “noses”.

At the same time, this might not even be an issue for Training Tests. It would be perfectly fine for the Training tests to be somewhat easier or more difficult than the actual graded test.

(3) I use this structure (2 Training/ 1 Actual) for every new topic I cover. Since these are multiple choice or True/False tests, they are automatically graded and the grade feedback is “instantaneous”.

Whether MC and TF tests give good assessments of what essay type exams assess is a question for another topic. However, I can’t refrain from emphasizing that it depends on :

(a) how well the MC or TF test is written

(b) what one is trying to teach (for example, such tests may be very ineffective assessment instruments if you are trying to teach writing skills, but quite effective in a introductory history course)

I’ll leave you with a TF question:

(Q) Of the two TF questions given below, Ümit will use only the first in a TF test given in an upper division philosophy course (circle True or False):

• Descartes was French (circle True or False)
• One of the common objections to the “Inverted Spectrum” argument presupposes a Verificationist theory of meaning (circle True or False)

Ümit confesses to having seen both given as test questions :))  Hail to the richness of life!

Regards

I am not quite certain that I know what you are asking. Are you referring to "pretest" given before you begin your research? It may be helpful to reword the process if you are trying to look at test-taking ability which of course is an entire different process. And then you may find a more appropriate literature search., particularly if it is done through the education research topic. Good luck GBH

Thank you for asking, Dr. Hurst;

My question resembles this question:

"Folks, my friends John, Mary, Jill....etc. etc. and Adam are all using the word "superpolyfragilistic", but they do not tell me what it means.  Do you know of anyone who can tell me the meaning of "superpolyfragilistic"?"

In my case, the words I am seeking a definition for are the words "retest" and "retake", as they occur in the titles (and body) of articles such as those listed below.  I could not find a definition in these, or any of the other articles I looked at:

                   "Middle School Students’ Perceptions of Having the Opportunity to Retest on Summative Assessments"

                   "Predictors of retesting propensity and score improvement"

                   "Redos and Retakes Done Right"

                   "Retesting in Selection: A Meta-Analysis of Practice Effects for Tests of Cognitive Ability "

                   "The effect of optional retesting on college students' achievement in an individualized algebra course"

The need for a clear definition arises from the fact that the expression "allowing students to re-take a test" is ambiguous.  It can be understood as described in (a), (b) or (c) in my original post.

Thanks and Regards

Don't give up.

Today's researchers are no longer able to read everything that may be of use. Instead today's researcher use software tools to mass download documents from search screens. Then they search the resulting document library. (E.g., they use a library-wide search for "definition" and "define" to pick up and list all the formal definitions, and search for "a resit is" to pick up the different informal definitions of "resitting". ) I use Qiqqa for this purpose because it also automatically finds themes for me. http://www.qiqqa.com/Register/70557

https://www.google.com/search?q=%22a+resit+is%22

I confess to getting exactly 50% in your T/F test. Is any other score possible?

It would be fun to do an inverted spectrum experiment. It parallels the experiment where the victim------ subject is given special glasses which invert the world. After a week the person sees and functions normally. After removal of the inverting spectacles, the subject takes a week to recover again. Today's technology allows us to use a colour camera and virtual reality goggles to invert all the colours. I predict that it will take more than a fortnight for the victim------ subject to adjust. In the middle of the experiment there will be some interesting arguments about the colour of things! (Like the arguments my wife and myself get into because she always uses two words to describe a colour, when I use only one!)

Sorry Ian, there is no partial credit--either 100% or 0% :)

Yes, technology. is helpful when it works.   I'll just share this with you in case you find it amusing--but be warned, it makes some of my colleagues cry.

I wrote 5 machine learning programs (I will call them “robots”) that “read” the text of one of my quizzes and get trained on this data.  This is basically a classification problem: each chunk of text is classified as incorporating a certain moral perspective {C, D, V, N}.  Each robot reads these passages and "notices" how they are classified.  This is “training” the robots. (to give credit: I used ScikitLearn's Python based software packages)

Then, the robots are fed the passages of the actual quiz without the classification information, and are asked to classify each passage.  Here is how they performed:

Robot using the Naive Bayes algorithm: 6/10

Robot using the K Nearest Neighbors algorithm: 6/10

Robot using the Neural Network algorithm: 8/10

Robot using the Logistic Regression algorithm: 9/10

Robot using the Logistic Regression algorithm (different settings): 9/10

All they do is "focus" on the passages and train to catch the clues that correspond to each category—and remember them. (Yes, they do not mind studying, the poor slave-minded things.  They must have been abused as children.) They even do not get a chance to ask questions while training, like my students do...

Here is a sample passage:

• “I, Ragnar Lodbrok, took an oath to become allies with King Horik.  I confess that I later found him to be a double-dealing crook.  He has already stabbed in the back two jarls (earls) he was allied with; and who knows, I might be next in line.  I know that some of you will say that I am duty-bound to keep my oath, that I should not take action against King Horik insofar as he does not do anything to me.  I say, in return, that I have other obligations.  One of them is to protect my family and people from such treachery—and this obligation is based on an oath I took much earlier than the oath for King Horik.  I cannot sit and wait for him to stab us in the back.”

And here is the feedback students get for this passage upon completing the quiz. (the robots do not get such feedback)

Truly a sophisticated leader who knows how to apply Duty Theory.  He underlines that he has multiple duties, and moreover, that sometimes, even a single duty (keeping promises) can prescribe conflicting courses of action.  He explains clearly why he will give priority to one narrower duty over the other.  A full grade to Jarl Ragnar if he were taking this course :)

The class average for the quiz beats two of the robots  (repeating pattern over multiple semesters)  :)

Regards

That's good fun. So "duty-bound", "oath" and "obligation" are hints that this is morality type "D" (for: "my Duty")?

You are very good at this; are you a robot?  Can't be, because you have a sense of humor.  But then, you could be simulating that... :)

{Not simulated!}