First of all, power analysis shouldn't be done based on pilot data (this is a common practice but it is actually widely discouraged) -- pilot data themselves are underpowered and thus do not offer a good estimate of either the magnitude of the effect or the variation of the effect, and thus aren't a good basis for power analysis. It's better to do power analysis based on more realistic estimates of the magnitude and variance you expect in your effect (based on a previous high-powered study or a review or meta-analysis of many studies). The purpose of piloting should for stuff like testing the feasibility of a study (e.g. to see if the presentation rate is comfortable for participants, to see if 'manipulation check' conditions work out if you have them, etc.), getting some example data so you can prepare your data analysis scripts to run later, stuff like that, rather than for power analysis.

For designs with many trials per person per condition, like what is typical in fMRI (and especially event-related fMRI, as far as I know), power analysis is extra tricky, because the stability of effects is affected not only by how many participants you have but also how many trials there are per participant. If power were just based on number of participants, analysis would be straightforward -- as you mentioned, you can algebraically calculate a Cohen's d based on the test statistic (t or z) and the sample size (N). But this isn't sufficient for typical fMRI experiments because the number of trials also matters: if you have two experiments with the same number of participants and the same % BOLD signal change or whatever between two conditions, then the experiment with more trials per participant will be more reliable.

As far as I know this is a much more difficult problem to solve. One way is to analyze your data with a mixed effect model (so you can include each trial of data, rather than averaging across trials to get an aggregate for each participant); from there, there are several methods available to estimate power for mixed effect models. To be able to do this well, you need estimates of all the variance components of the data, which is why it's much easier to do if you have a real dataset available (unlike the case for a simple t-test or ANOVA, where all you need is a Cohen's d or partial eta squared from some previous study/studies). What I find easier is to do it by simulation [which is also how some approaches to estimating power for mixed effect models work].

Here is a script I use for it: https://politzerahles.shinyapps.io/ERPpowersim/ . This script is for ERPs but I imagine would apply similarly to fMRI data. There are several limitations/challenges. First off, like the approaches mentioned above, you need estimates of all the variance components of the data (how much things vary within participants, across participants, etc.), which is often most easily obtained from a previous dataset. Secondly, this particular tool I use focuses on a targeted comparison for one time window and one region; if you want to test across many regions and need to worry about multiple comparisons, or you want to do some other methods like conjunction analysis or PPI or all kinds of stuff, this might not be directly applicable.

Dear Minwoo, as Stephen said, sample size estimation not necessarily needs to be done taking in account pilot data - often they are not representative of the population you want to infer something.

I usually say that the settings for a sample size estimation - a) the minimal effect size you expected to detect; b) the maximal standard deviation you expect to find in your sample; c) the power you want to detect type 2 error; d) your threshold for type 1 error - are totally an investigator-choice (examples for a randomized clinical trial).

You can take a bunch of evidence from various sources (past studies in similar settings; the minimal clinical significant result you might consider to justify the existence of your experiment; meta-analyses; personal beliefs - i.e., as the same of an informative Bayesian priori) and estimate samples in which you think that are adequate for your statistical and feasibility purposes.

That said, for a fMRI study (that's not my field, but the thinking is the same), beyond the metric to assess your results and your outcomes, you should also consider the study design. If it is a cross-sectional study, you need to take in account other things; if it is a diagnostic study, the same; and so on.

Please make this thinking-exercise that might help you to define the best settings within a mix of evidence and literature knowing.

Thanks,

LH

Thank you Stephen and Lucas for your thoughtful comments (and also for the code!!). I also appreciate the point that we should not rely too much on the data from pilot studies which are typically underpowered themselves.

Yes, I agree that fMRI power calculation takes extra steps to carry out than what behavioral data analysis would typically require. While I understand the difficulty, I'm curious about how researchers typically carry out this process (or acquire relevant estimates) in practice, given that actual data sets (that match your experimental design and variables of interest) that allow you to directly calculate effect sizes are not readily available.

By the way, I came across Poldrack at al (2017) which essentially suggests a similar strategy as suggests by you guys. It utilizes a huge dataset available from human connectome project. Please check it out if you are interested!

http://fmripower.org/

Article A power calculation guide for FMRI studies

Thank you for sharing this. I had read the article as I posted the question, finding that fMRI power operates on a pilot dataset and I was looking for ways to circumvent it. Still, I think that the paper is very informative in that it reviews several approaches that have been proposed by different groups of researchers. I really appreciate your suggestion!

FYI, I'm sharing another, web-based, toolbox that might be helpful:

http://neuropowertools.org/neuropower/neuropowerstart/