As you point out there are different ways to treat the multiple measurements. One way that depends on how you take the multiple measurements is called longitudinal models. You can find a set of notes written by Prof Marie Davidian, a real expert in the field at:

https://www.stat.ncsu.edu/people/davidian/courses/st790/notes.html

Start with repeated measures and then go as far as you want into the subject. This is really useful stuff for design and analysis of experiments. Best wishes, David Booth

As you point out there are different ways to treat the multiple measurements. One way that depends on how you take the multiple measurements is called longitudinal models. You can find a set of notes written by Prof Marie Davidian, a real expert in the field at:

https://www.stat.ncsu.edu/people/davidian/courses/st790/notes.html

Start with repeated measures and then go as far as you want into the subject. This is really useful stuff for design and analysis of experiments. Best wishes, David Booth

As to how this can be put into power analyses:

A useful tool for power-analyses is the free G*Power software. You can just play around with different numbers of measurements to get a feeling for their influence on the test's power.

You can find the download, the manual, a tutorial and scientific background under: http://www.gpower.hhu.de/

Hi Jim,

Multiple measurements will generally increase power. The influence of outliers will be diminished, especially for highly variable measures like response time.

For some measures, (particularly continuous measures) you run the risk of oversampling. But, it seems like you primarily use discrete measures.

The other considerations are methodological. You have to weigh the trade off between increased power and order/practice effects, which have the potential to adversely influence your validity.

The only time that you would treat the repeated measurements individually would be if you were interested in change over time. In which case, you would treat trial order as a within-subjects variable. Otherwise, you will artificially inflate your power. I have always averaged across repeated measurements, but it looks like David suggested some intriguing, more sophisticated approaches to this issue.

Best of luck!

Brandon

Thanks to David Eugene Booth, Martin Westhoven and Brandon Thomas for your responses, these are all really helpful.

I guess the crux of my question is how you can factor in the potential increase in power obtained by taking multiple measurements (and then averaging these multiple measurements) when carrying out an a priori power analysis to determine the sample size required for an experiment.

For example, consider I am carrying out a simple experiment with a single, two-level between-subjects factor, and I plan on using an independent t-test to assess the difference in responses between two groups. I anticipate finding an effect size of Cohen's d = 0.5. How can I assess what size sample I require if I plan on taking 3 measurements of the same DV for each participant and averaging across these 3, as opposed to if I were taking only one measurement from each participant.

Martin Westhoven , GPower is a great piece of software. If I use it to assess the required sample size for an independent t-test, I can't see whether I can account for the number of measurements I might take from each participant. Perhaps I am looking for something that doesn't make sense or is not possible to factor into power analysis?

Thanks again for the responses.

Jim

If I am not entirelly mistaken, you're not really looking at independent measurements anymore, when using repeated measurements. If you average across the measurements, you effectively drop the information about how many measurements were taken. Also, a bias could be introduced to the averaged value through the order/practice effects Brandon Thomas already mentioned.

So while averaging allows you to still use independent t-tests, you have to accept some drawbacks.

You could try an ANOVA for analysis, though. Repeated measurements ANOVAs are built for this kind of mixed design. Last but not least, G*Power factors in the "number of measurements"-parameter for RM ANOVAs / MANOVAs, so you can compute the required sample size with relative ease.

The power computations for RM ANOVAs would also be my starting point to a deeper understanding of the mathematical background of repeated measurements effects on required sample sizes.

Hi Jim Uttley , I realize this response is coming maybe a bit later than you may have wanted, however I hope it can still be helpful.

For multiple measurements per person, you can calculate power using GPower just like you would for an OLS regression or non-repeated measures ANOVA (using GPower3, assuming you are familiar with that software) but adjust sample size for dependence between measurements. This method is appropriate for 2 level cross-sectional & repeated-measures models, which to me sounds like it fits your case. To do this, you would calculate what's called "Neffective." Neffective = N / (1+[n-1]*r), where N = total number of data points, n = group size, and r is the intraclass correlation (ICC) between measurements(or timepoints). In your case it sounds like group size corresponds to number of people in your study. In case you are unfamiliar with ICC, here's a web resource that's fairly well-written: http://www.theanalysisfactor.com/the-intraclass-correlation-coefficient-in-mixed-models/. And here's a link on how to calculate it in SPSS (Article Intraclass correlations: Uses in assessing rater reliability

Its interpretation is such that the value of the ICC corresponds to the % of variance between groups (between your 2nd level, or person level), whereas whatever is your finest level of measurement (trial, time, etc.) is the leftover percentage of variance to be explained. To make this more concrete, for repeated measures design (your case), assuming a multilevel model approach where trial is at level 1 and person is at level 2, an ICC of .41 would indicate that 59% of the variance in the outcome is within-people and 41% of the variance in reading achievement is between-people.

Once you calculate Neffective, you would simply use this as the sample size in GPower3 when calculating sample size in order to provide an accurate assessment of power. If you use Neffective, the appropriate citation would be: Kenny, D. A., Kashy, D. A., Cook, W. L., & Simpson, J. A. (2006). Dyadic data analysis (Methodology in the social sciences). New York, NY: Guilford. Along with the citation for GPower3. 

Another option if you go the Multilevel Modeling route that I'm aware of is using PINT (power in two level designs) software. It can be accessed via this link: https://www.stats.ox.ac.uk/~snijders/multilevel.htm#progPINT. Unfortunately, this software cannot handle models greater than 2 levels, but might be appropriate for the analyses you would like to perform. Its a little more complex than Neffective. There is more information on Snijders website.

Finally, conceptually, I would almost always want my research design to match the model used - so I would recommend using each of your measurements, rather than averaging over them. This will not only increase your power but also give you the most accurate estimates of parameters in your statistical model, since your model will exactly match your experimental design. And now you (hopefully, if I was clear enough) have a method to account for the interdependency of measures, and still get an accurate estimation of statistical power.

Hi Jim,

I agree with the answers above regarding your original question. Below is a suggestion regarding your second question (i.e.,"For example, consider I am carrying out a simple experiment with a single, two-level between-subjects factor, and I plan on using an independent t-test to assess the difference in responses between two groups. I anticipate finding an effect size of Cohen's d = 0.5. How can I assess what size sample I require if I plan on taking 3 measurements of the same DV for each participant and averaging across these 3, as opposed to if I were taking only one measurement from each participant")

The short answer is, averaging a score of repeated measures (no matter how many measures you give) has no effect on power because you still only have one outcome score per person. Treating each measure as a unique score increases power, but creates an issue where you violate independence and requires an analysis with a within-subject factor or other correction for dependent data.

One way to think of it is in terms of your outcome. In the above case, you take 3 measures, BUT, if you average them, then you only have one outcome measure per participant (i.e., their average score). So, even though you are taking multiple measures, when you average them, you go back to having only one measure per participant... in other words, you still only have one outcome score per person (you only have one row per person in your data set). If you are averaging multiple measurements, the number of measurements you take has no effect on power because you are reducing them down to one score when you take the average.

For example, if you took 5 measures on 10 people and calculated the average, you would have only 10 scores on your outcome (i.e., one average score for 10 people). If you took those same people and measured them 100 times and averaged the score, you would still only have one outcome score per person (again, their average score). In both cases, whether you administer 5 or 100 measures...if you calculate their average score, you only have one score per person. As long as you are using their average score as the outcome, it will have absolutely no impact on power if you measure them 5, 100, or 1000 times. The only way the number of measures per person could influence power is if you treated each score as a individual score on your outcome variable.

Using the same example, if you treated each measure as a unique score, you would have 50 data points in the first example (5x10) and 1,000 in the second example (100x10). Obviously, 1,000 data points will have much higher power than 50.

If you want to do a between factor, independent sample t-test as you mentioned, you can only use their average score. If you treat each score as unique, you no long have independent data, as repeated measures of the same person creates dependency (not to mention fatigue, learning, etc. that was mentioned above) and you violate that assumption of an independent sample t-test. If you treated each measure as a unique score, you would have a within (or mixed if you had other between factors) design and would need to model the within component (for example, using a dependent sample t-test, a repeated measures design, or a mixed if you have both between and within factors).

I hope that helps, best of luck!