Dear James, it's a bit difficult to answer your question, since you're not really clarifying exactly what your question is. Therefore please view the following as more of a comment. You write: "I haven't done any kind of p-value statistical stuff on our sample, but am I missing something?", well if you have a hypothesis, then you need to compute a p-value to estimate the validity of your hypothesis and if you're worried about if you have enough samples to detect a given difference, then you need to take a closer look at power calculations. Hope this helps!

I guess mainly I'm just not sure how adding white light measurements to magnetograms would increase the sample size. Even if we included white light observations in our analysis, we'd still be looking at the same 1075 active regions. 

Perhaps the issue is not the size of the sample but how the sample units are selected. If the selection method is not random, it is not possible to do proper inference (whatever the sample size).

Paul, 

Thanks for the compliment! However, the paper this question thread is appended to is not mine. I think your quotation comes from their paper, which cites mine and says that my results may have been unreliable due to small sample size. My sample size was 1075 active regions that had very few selection criteria applied (to Ivan's point, e.g., active regions within 30º of solar disk center as observed from Earth, which should not impart any systematic bias to the overall sample since there's nothing special about Earth's position). We ended up with 98 X class flares, 1139 M class flares, 4932 C and smaller class flares, and 410 active regions that had no associated flares at all. 

Christopher, you've taught me something new! Statistics remain a fantastical realm to me. 

Indeed, that's the right abstract. We didn't calculate p-value because it just never occurred to us. It's not something people in our field typically do; I just got exposed to it pretty recently from judging at the Colorado Science and Engineering Fair. The statistical method we used is called superposed epoch analysis, which is a pretty niche tool I haven't seen outside of solar physics. It's simple to implement and designed to pick a weak signal out of many similar events, which of course requires a whole lot of samples. 

One point nobody has yet made is that the phrase: "these studies may be unreliable because they are not based on statistically significant samples" is nonsensical. There is no such thing as a statistically significant sample. Statistical significance is a property of the data. It speaks to whether an observed relationship is likely to occur by mere chance. As such, it is a function of the interaction of effect size and sample size.

As you say, the so-called exact sciences rarely employ significance tests, partly because the observed effects tend to be so large that it would be silly, partly because a precise point estimate yielded by a theory is more impressive than a mere "p

Your question is not clear.

If  study is based on any sample, and if you want to give inferences about population than you have to check it.  Statistical significance tells to whether an observation is likely to occur by mere chance or it is true for whole population. For that you have to perform a test. p value is used for decision purpose. if you are not using 'Satistical significant' , there is no need of estimating p value 

@Abhay, sure but I agree with Stephen: p-value is typically only used in the social sciences, not the hard sciences. This is the latter and even though my sample is definitely statistically significant, I don't think a p-value is the right way to prove it. At the very least, my results were much more statistically significant than the paper this question is attached to. Their sample was 1000. 

According to me, if sampling design is suitable and sample present whole population, that means   sample is 'statistically significant samples'.