Hey Bernd,

Basically, the main advantage of having only one tracer is to avoid having several people spending days delineating specific structures on each slice for each subject.

But, especially in the publication process, you might be asked to provide metrics of intra- and inter-rater agreement (usually ICC, Intraclass Correlation Coefficient) to give an idea of the reliability. You can imagine a situation where two individuals are very consistent with themselves (high intra-rater reliability) but strongly disagree with each other (bad inter-rater ICC: one of the tracers is always more "generous" than the other: he includes more voxels in each segmentation and gets larger volumes). That would show that your measurement is not really reliable (many the guidelines you are using are not precise enough or you dont follow them as rigorously as you should).

More generally, knowing that no measurement is going to be perfect, having several measures for each subject will also allow you to get a final (mean) value that is probably "cleaner", ie more likely to be closer to the "real" value that any individual value.

Hope that helps! Good luck with your study

Renaud

Thank you, Renaud !

I'm finding similar issues as stated by Renaud above, where the protocol needs to be exceptionally precise and rigid. Doing inter-rater by correspondence alone does not work. The second rater needs to be guided visually, at least for a few scans before they go off and trace the inter-rater sample. This is just such that they begin to make similar judgement calls as the first rater.

I would refer you to Dewey et. al. (2010) Where they observed an strong intraclass correlation between manual and Auto-assisted Manual (AAM) techniques in all structures measured (Cronbach’s α>0.90). In my experience so far with inter-raters an ICC between 0.90 and 0.95 is extremely good even between manual raters. AAM seems like a decent pay-off and saving a lot of time, and money. However, in presence of serious pathology I would not suggest it, as well as if you have a sample of children.

(Dewey, J, Hana, G, Russell, T, Price, J, McCaffrey, D, Harezlak, J, Sem, E, Anyanwu, JC, Guttmann, CR, Navia, B, Cohen, R & Tate, DF 2010, ‘Reliability and validity of MRI-based automated volumetry software relative to auto-assisted manual measurement of subcortical structures in HIV-infected patients from a multisite study.’, NeuroImage, vol. 51, Elsevier Inc., no. 4, pp. 1334–44.)

I conduct manual volumetric tracing in my lab and agree with Renaud and Steven. We are always negotiating these issues. I would additionally recommend not having too many tracers involved in a particular study, when possible only 2. Despite robust protocols and clarity of the ROIs, every brain is so different that there is going to be some measurement error introduced while tracing, you want to keep it under control by having less tracers.

A recommendation I received from a colleague, which has proven to be useful in the long run, is to have a separate brain just for training purposes, in which both tracers can seat down and manually trace the structure while discussing potential areas of disagreement, fuzzy areas and areas that may require protocol adjustments. Then, on a second brain each tracer manually isolates the ROI and ICC are obtained. Once you have a strong sense that the tracers are ready to work independently, there is going to be a higher probability of success in your ICCs.

Some reviewers will also ask not only for inter-rater reliability but also for intra-rater reliability, so the less number of tracers you have, the less time everyone is tracing and re-tracing!

Thank you Steven and Mauricio ! I especially like the idea of doing a training on just one brain first. I must admit I have no idea how many different protocols for manual tracing exist. Therefore the EADC-ADNI effort to harmonize protocols sounds really promising.

I have been trained extensively in manual volumetry, and have traced dozens of regions on hundreds of brains. I'm unclear regarding Maricio's recommendation, and although I agree that you might not want more than 2 tracers on any given brain, I think that tracing a large data set using >2 tracers is perfectly acceptable as long as they are all reliable with one another on all regions being assessed. However, this necessitates a great deal of training before a team can trace, and is thus cost prohibitive for many labs in terms of time.

ICC(2) and ICC(3) values (see Shrout & Fleiss, 1979) for inter-, and intra-rater reliability should ideally be .90 or above for all pairs of tracers. In addition, an important point I do not believe has been mentioned is that having two tracers per brain, tracing in randomized order across all slices is a way of distributing individual bias across brains and minimizing the bias of a given rater in any one brain. If you only have a single rater (sometimes the case) then you can test yourself by tracing a given set of ROIs/brains and re-tracing it after some reasonable delay (e.g., 2 weeks) and comparing the values using the ICC(3).

Furthermore, I strongly recommend using far more than one brain for training. This is primarily because what you need to train on is not only in the perceptual consistencies of tracing tissue interfaces similarly with other raters, but also because the greatest variability can be introduced when one is faced with a decision point in manual tracing following a set protocol across raters. Thus, by practicing and training on multiple brains, one can have a better understanding of the specific anatomical and morphometric rules for a given ROI.

I agree with the recommendations Andrew makes in regards to training. I should have added "at least" one brain. When possible during training, using several brains does broaden the scope of experiences with the variabilities we all know the brain offers. In my lab we use several brains during the teaching of tracing procedures, which students pass with ICC scores >.90. After they are well trained, one brain to study some specific ROIs (e.g., corpus callosum) has sufficed. For some ROIs more than one brain will be needed to achieve the desired reliability.

Thank you Andrew for providing the Intra-Class correlation cut-offs. Another reference that has been useful to me is the McGraw & Wong, (1996), Forming Inferences About Some Intraclass Correlation Coefficients (in Psychological Methods), which has been useful for me to understand the ICC.

I am excited there is ongoing interest in these methods!