I think one needs to have some idea of what the learning effect actually entails. If the effect is simply that each person's score improves by some set amount, then the test-retest reliability coefficient will not be biased. Remember that the coefficient is measuring the correlation between scores on the two occasions. A change in the mean of the scores from pre-test to post-test will not affect reliability in of itself. However, the coefficient could be biased:

So there could certainly be biases present, but I'm not sure that they are specific to memory tests. (1) could certainly apply to other cognitive ability tests if administered using the same items, and (2) could even apply more widely to things like self-report personality scales.

Leaving a long gap between testing periods would not really help. For a test-retest coefficient to be an accurate estimate of reliability, there should be no change to the underlying trait (i.e. memory ability). If a long delay is enforced, subjects' actual memory capacities will change. That kind of strategy can be useful if the goal is actually to assess changes in memory for an individual participant, but that isn't an assessment of test-retest reliability.

First, measuring "reliability" presents no additional problems if using something like Cronbach's alpha compared with personality tests (though with all the usually caveats about the limitations of alpha).  Suppose my memory test involves showing subjects 100 word pairs and then at test I give them one word and ask for the other. The result for subjects is a bunch of 0s and 1s. If one wants, you can calculate alpha with this (KR20) or run an IRT or whatever you like. In fact, many academic tests can be thought of as memory tests, and reliability is often reported.

Stability based on test-retest designs is something different, and if you are giving the same test twice you are not testing exactly the same thing the second time if the person has learned. If you wanted to measure stability of memory processes then you would want to use different items. Giving the same 100 twice would be looking at something different. Still, having the same type of test twice provides some practice, so even with parallel forms the second testing is not identical, but usually that is all that practical to do.

What do you want the stability measures for?

The problem with all these old methods from pre-computer psychology is that they fail to recognise the importance of measuring the same variable, e.g. response time, for the same subject over an extended period of time. Then you can use more modern time series methods to discover the changes that occur in performance. In most cases these changes involve nonlinear dynamics, a situation that cannot be assessed using linear correlations.

I think one needs to have some idea of what the learning effect actually entails. If the effect is simply that each person's score improves by some set amount, then the test-retest reliability coefficient will not be biased. Remember that the coefficient is measuring the correlation between scores on the two occasions. A change in the mean of the scores from pre-test to post-test will not affect reliability in of itself. However, the coefficient could be biased:

So there could certainly be biases present, but I'm not sure that they are specific to memory tests. (1) could certainly apply to other cognitive ability tests if administered using the same items, and (2) could even apply more widely to things like self-report personality scales.

Leaving a long gap between testing periods would not really help. For a test-retest coefficient to be an accurate estimate of reliability, there should be no change to the underlying trait (i.e. memory ability). If a long delay is enforced, subjects' actual memory capacities will change. That kind of strategy can be useful if the goal is actually to assess changes in memory for an individual participant, but that isn't an assessment of test-retest reliability.

Test-retest reliability is in any case better than internal consistency based on CA or other statistial measure. I may also add for clarity that it is possible and necessary to use tests with high reliability measures. Using computerized, or rather experimentally valid, methods based on experimental research will naturally improve both reliability validity in a more substantial sense.

Based on my experience as a neuropsychologist at a memory clinic, and a long history of test construction, it has been rather discouraging trying to convince collegues that many of the tests used for memory/attention assessments are really obsolete.

Recently we published a study, Memory factors in Rey-AVLT: implications for early staging of cognitive decline, 2014, Scandinavian Journal of Psychology. One may say that learning effects and memory effects may be found using this old test or other similar tests, eg California VLT. Therefore the examining psychologist may be content in showing differences between healthy controls and MCI-patients using them. However, based on the huge amount of memory research from the 60s until today, I still would not advice examiners to use such an obsolete test in 2014. The reason why psychologists still use old fashioned tests like the Rey-AVLT is probably that the two main fields in psychology (experimental and differential psychology) have drifted to far apart. 

What is needed is a marriage between experimental and differential psychology in order to use the latest, and probably best, ways of measuring memory (or other cognitive capabilities). It is still OK, in my opinion, to use paper and pencil tests to a degree, but in most cases computerized tests using modern and experimentally valdid procedures are definitely better in order to improve neuropsychological diagnostics or whatever assessment is needed.

Just take, as an example of improper administration of tests, how words are presented and checked for memory during the Rey-AVLT: the examining psychologist uses his/her own voice, used his/her own speed of presentation, and is concentrated on checking words during the test phase. Such improper procedures would be regarded as scandalous in any experimental session today, when computers are available. In order to collect more interesting data, it would also be aviced to use computer testing, especially compared to what the psychologist is able to check during a memory examination session based on paper and pencil.

Note: a test may show a high degree of reliability, and still be of less value regarding specificity. Our analysis of Rey-AVLT showed that primacy, recency and interference as well as learning are involved during the performance of the test. If the psychologist strictly follows the manual, using the lumped scores the test shows high reliability in the sense that a low-scorer will score low a second time, and a high scorer will score high.

But the lumped score of the obsolete Rey-AVLT test, is less interesting than a procedure that takes care of experimentally well established measures. To continue the use of old fashioned testing procedures in diffential psychological settings is, in my view, is to disregard the hard working experimental and scientific psychological community.

I think Matt Williams gave the most useful answer. As long as "learning" from session to session is fairly consistent across individuals, the test-retest correlation will be robust. But you would want a reasonable interval between administrations (at least weeks or even a month) to ensure that most participants didn't already remember the stimuli. (One nice idea would be to ask them, at the beginning of the second session, for any words they still remember! Then that would be in there as a control.)

The other approach, of course, would be to use a test with an alternate form. This is justifiable to the extent that the two forms are really equivalent. Establishing their equivalency is just another type of reliability study. You administer both in the same session or a day or so apart (counterbalancing order of presentation). The correlation coefficient is the reliability coefficient. It should be very high if you want to argue that the forms are equivalent. And you need to establish that the mean scores are similar. Ideally, the distribution of scores will also be similar. If this has been done, then you already know the reliability of the measures, but can go on to check on test-retest reliability if you want.

I have always been disturbed by how few neuropsychological tests actually have equivalent forms, given the importance of re-assessing patients. If it can be done with achievement tests and language measures, why not memory or executive function tests?

Computer-based assessment naturally has its advantages. On the down side, if you are a practicing clinician, you don't get to actually observe and interact with your client. I have to say that no clinical neuropsychologist I ever heard of simply interprets the "total score" from the RAVLT or CVLT. There are all kinds of measures contained in these instruments, including learning curves, interference effects, recognition benefit, and so forth. The problem with contemporary experimental psychology is that the research foci are so refined. We already know most of what there is to know (descriptively) about phenomena like retroactive interference, so nobody is going to conduct research on it. It's similar to the difference between particle physics and repairing a bridge.

I am glad to hear that clinicians of today don't strictly follow the Rey-AVLT manual.

Regarding the use of computers during examinations, my experience is that they do not interfere at all with the the quality of patient-psychologist interaction. In many cases a face to face situation during testing may actually increase test anxiety for some patients, while other patients may find it motivating and thus increase their performance. Such personality factors should, in the ideal case, not interfere with test performance. Sitting beside the patient during testing reduces such obvious errors of measurement. This is common social psychology.

It is much easier for you as a clinician to observe the patient during a computerized test, than if your attention is divided between observing patient performance behaviour and your own scoring actions. A computerized test is, from my experience, also more stimulating. An often made comment from patients is "This is really fun!" This, however, may be due to the clientele of a memory clinic with relatively old patients. 

But there are other obvious advantages using computerized versions or methods developed by experimental psychologists such as similarity between presentations.

This is true not only for memory testing, but for all kinds of testing. In the middle 80s my supervisor, professor Åke Hellström, developed a version of his TOE procedure (Time-Order-Error) for an Alzheimer's treatment experiment, earlier used in experimental perceptual psychology. That procedure is not possible to use without a computer. I used the TOE-test with patients diagnosed with Alzheimer's disease. That type of test has focus on performance than a typical memory examination test, and is therefore, in my opinion, less stressful. Other perceptual tests based on illusions are also adviced by me. 

I may have informed the community about my test Male Faces before, but I take the opportunity in this forum to mention that it is completely impossible to collect similar data from a paper and pencil version of that memory test (or other recognition tests) in a comparably reliable way. The Male Faces test was coded in the early 90s using the HyperCard program and later recoded for both Mac and PC using the REALBasic language (Xojo) in order to arrive at better precision of RT measurements.

Using a Mac or a PC you may collect both reaction time data for each trial of Male Faces (42 trials) in addition to standard results (% correct, etc). The test development is described in our article Paradoxical Prosopagnosia... published by the open access journal AAD, Advances in Alzheimer's Disease. It may be noted that Male Faces has a normal distribution, so far, and the high score is below 100% correct.

This discussion may encourage me to have the test packaged for distributionin order to make it available also to other communities in the neuropsychological world.

Note: the selection and preparation of face photos was the most time consuming part of the project, and actually started together with the late Stanislav Dornic already in the early 80s for a much different project. In the AAD article other memory tests, that I coded at the same time as Male Faces are also described, one of them that is used to fill the retention interval. 

I liked the point of view of Stephen Joy, and I would conclude this:

Reliability has two major components:

Internal consistency: refers to the degree to which different items, parts or components of a test measure the same thing.

Temporal stability: refers to the extent to which a measuring instrument will yield the same result in different specific measurements by measuring an object or subject that has remained unchanged.

There are various methods to evaluate the temporal stability, that is probably what you are looking for, but the test-retest is a good choice for those issues that are not influenced by learning or repetition and that remain constant within a ample time, as Matt Williams said (eg. personality), however it is not advisable for intelligence or memory tests, specifically those such as vSRT (Buschke), RAVLT (Rey), CVLT, as Sven-Erik Fernaeus said, requiring to memorize a list of words, in this kind of test the effect of repetition is crucial,  and has been found that people can remember some of the words in the list until one year after application.

Anyway all ratings (X) measure a given construct (V) and also measure that produced by the error (E), so you will never find a completely reliable test, but it will always have a degree of error included in those methodologies or computer based testing:

X = V + E

Other ways to study the reliability, possibly more suitable for this type of testing is the split-half or constructing alternate forms, but you need to see in which field are you going to apply it: experimental or clinician, has Sven-Erik Fernaeus, said booth fields may be splited, but is our duty to unite it again.

Thank you all for your insightful comments. I will take them into account in my research.

The focus of my question was mainly related to the way of presenting the psychometric properties of a memory test for clinical use. When studying widely used tests, usually we don't have alternate versions.

It seems that there are at least three main approaches:

However, even using theoretically equivalent forms there are some limitations to the study of reliability. 

In the position paper "Official position of the American Academy of Clinical Neuropsychology on serial neuropsychological assessments: the utility and challenges of repeat test administrations in clinical and forensic contexts" Heilbronner et al. (2010) the authors say:

"Practice effects are especially challenging—yet important—to consider in serial memory testing because memory measures have been found to produce particularly poor test–retest reliabilities (Dikmen et al., 1999). With the knowledge that memory measures generally have lower reliability, and that subsequent exposures to the same memory stimuli represent additional learning trials, Dikmen et al. (1999) recommend use of multiple memory measures and reliance on composite scores, rather than subtest scores. Several studies have shown that scores may be enhanced on retest even when the test items are different (Benedict & Zgaljardic, 1998; Wilson et al., 2000). This can occur even with the use of alternate forms, as the examinee benefits from knowing how to approach the task more effectively (i.e., the examinee has acquired a test-taking response set or strategy or has become more familiar"

In my opinion the most relevant point is to be aware about the implication of serial testing.

Thank you so much for your interesting contributions.

I will have to object towards the Dikmen et al (1999) recommendation to use composite scores, because memory is not one-dimensional. I must admit that I never heard of Dikmen before, and I am quite surprised to hear that memory tests have low reliability. It is definitely not my experience. I suppose it depends on what test we are talking about. I will immediately try to find the Dikmen et al study.

In cases with 3 months or more between sessions including several tests, not only episodic memory tests, there would be small and often insignificant learning effects if we take into account a typical situation, i.e. memory examinations in order to assess memory capacity in suspect MCI, Alzheimer's disease, etc. However, why not use such learning effects as an additional measure: if healthy controls may show learning effects at widely spaced sessions (+ 3 months), and if MCI or AD do not show any learning effect, this difference (if significant) in itself is a measure to take care of instead of regarding learning effects as measurement errors or signs of low reliability.

I take the opportunity to mention a test, Months Forward/Backward, that we used in a study, Östberg, P., Fernaeus, S-E., Bogdanović, N. & Wahlund, L-O. (2008). Word sequence production in cognitive decline: forward ever, backward never. Logopedics, phoniatrics, vocology, 33, 126-135. The Months test would be labeled a test of semantic memory, because we usually regard the knowledge of the order of months a general knowledge easily retrieved. However, based on BB Murdochs account of memory (Memory: theory and data), that knowledge may be based on a combination of associative and serial memory resulting in a representation of the list (or table) of month number, and month name in each row. In the forward mode the testperson says the names of the months in the same manner as he/she would say the names of the days of the week, ie as an overlearned list of items - or as we assumed above as a part of semantic memory. When asked to say them in reverse order the testperson has to rely on his/her knowledge of the calendar, not only a habit to go from January to December in a somewhat automatic fashion.

My point: the Months Test F/B does not use any "items", we may in fact say that the concept "test items" is irrelevant here. Still, the test has good reliability, and discriminates between SCI and MCI as well as between MCI and AD. I also regard the Months test extremely useful due to the speed of administration.

Other memory tests of a similar kind is a type of test that I never published, I only used it as a starter: the patient was asked to tell me about the last time when he/she was a) bying shoes, b) seeing a movie, c) going to the dentist. The score was RT for each item.  Preliminary results showed that RTs above 10 seconds were associated with memory decline and often dementia. It would be interesting to see a study using that type o test, because it comes very close to measuring basic aspects of personal memory.

In a traditional statistical sense it might be regarded as inherently low in reliability, because there are large differences between individuals regarding retention intervals. However, retention intervals are not always very important: in an - also not reported yet - study on memory for cross word solutions based on crypts the subject made 100 trials with different crosswords varying retention intervals (15-800 minutes). Study time for each solution (beta=.27) as well as trial number (beta=.81) predicted the memory performance for the 26 letter-digit combinations (R=.75). Retention interval did not add to the prediction and was insignifiant. At about trial 30 the subject arrived at an asymptote of about 95% correct including many trials with 100% correct. Due to the time consuming (2 years of data collection) procedure I would not recommend it as a test though! However, any problem solving task involving letter-digit solutions may be used, and data collection may be done letting the patient do the task at home every day, and using email to communicate with him/her, and engage the spouse as an assistant.

My point is to remind memory testers that Ebbinghaus methods may age in grace, although those methods were introduced about 130 years ago. 

Memory is indeed not a unitary function. For example, in the WMS-IV (Wechsler, 2009) - a paper and pencil test battery – both working memory tests and episodic memory tests are included. Moreover, the episodic memory subtests consisted of two different auditory and two different visual-spatial tasks (in order to investigate modality-specific effects); each test examined direct and delayed recall / reproduction (in order to investigate consolidation of memory items) and had also a recognition procedure following the delayed recall task (in order to investigate active retrieval of memory items). Working memory (two different subtests) is examined in the visual modality, while auditory memory can be measured in the auditory modality. In contrast to episodic memory, working memory is an important component of executive functions (see e.g. model of Baddeley, 2003). Shortly, a test battery deliver a profile of memory weaknesses and strengths, with norms of the memory tests and memory indexes and norms for different performances between memory tests and memory indexes.

To give an idea of test-retest reliabilities with a mean test-retest interval of 23 days, the correlations of the different memory indexes ranged from 0.79 to 0.82, and the correlations of the different subtests were lower and ranged from 0.64 to 0.76. Learning effects were expressed in terms of Cohen’s d.

Internal consistencies are very high (> 0.90) for the memory indexes, and somewhat lower for the memory subtests (0.74 – 0.96).

You can use computer tests, but computer tasks (or tablets) could be especially difficult for older people.

Anke, I really appreciate your insights and information about the WMS-IV. My experience, based on an earlier version of WMS is that these tests captures many parts or dimensions of memory in a highly reliable way. Unfortunately we have not published any analyses focused on WMS. We have taken them for granted, because WMS is such a robust battery, not problematic in any way.

Our preliminary data has repeatedly shown that the story recall test (WMS-Logical memory) is among the best predictors of dementia, and impairment in that test correlates significantly with both lateral and medial temporal lobe volumes in samples of consecutive patients and healthy controls examined at a memory clinic. Correlations with volumetric scores ranges from .37 to .51. Note, due to measurement errors we excluded stanine 5 of the volumetric scores according to the extreme grouping paradigm. Note also that correlations with lateral temporal lobe volumes tended to be higher than those with medial temporal volumes, perhaps due to less measurement errors for the lateral volumes.

Regarding the use of computers, as I said above, we have used them since the 80s with demented patients, mostly above 70 years of age. It is absolutely possible to code memory- or attention tests that are easy to respond to by older participants. Using iPads for instance, response buttons may be designed large enough.

I see no immediate need, however, to make a computerised version of WMS-Logical memory, and copyrights would not allow it anyway.

I have used recognition tests, as i mentioned above, with older people than me (!) with left and right arrows as response buttons without any problems. However, today I wold recommend using iPad or other tablets with graphical buttons instead. In cases where the patient has no experience with such response media, the learning curve is very steep. It only takes a couple of minutes of practice for an older patient to be accustomed to this form of response.