I agree with Ronán Michael Conroy's statement, "Effects that show up in small samples are big effects." However, they are also more likely to overestimate the true effect size in the population. See Button et al. (2013), for example. Here is their abstract (highlighting added):

"A study with low statistical power has a reduced chance of detecting a true effect, but it is less well appreciated that low power also reduces the likelihood that a statistically significant result reflects a true effect. Here, we show that the average statistical power of studies in the neurosciences is very low. The consequences of this include overestimates of effect size and low reproducibility of results. There are also ethical dimensions to this problem, as unreliable research is inefficient and wasteful. Improving reproducibility in neuroscience is a key priority and requires attention to well-established but often ignored methodological principles."

https://www.nature.com/articles/nrn3475

Article Power Failure: Why Small Sample Size Undermines the Reliabil...

Finally, the the two 2005 Lancet articles by Schulz & Grimes remain among the most thoughtful pieces I've seen on the topic of multiplicity. The second article includes a nice discussion of sub-group analysis.

Article Schulz K, Grimes DMultiplicity in randomised trials I: endpo...

Article Multiplicity in randomised trials II: Subgroup and interim analyses

HTH.

The trouble with having a rare disease is that science knows very little about you! Of course you can carry out subgroup analyses, but you should report confidence intervals carefully so that the imprecision of your study is also communicated.

I once worked on what was then the world's largest series of patients with a particular disorder – sixteen, as I recall. Still, we found some useful things.

Of course you can. Whether such analyses can lead to any meaningful conclusion is a different matter altogether. With such small samples, the signal to noise ratio tends to be very low and you are very likely to find values drawn from the extremes of whatever true distribution your data follows. As Ronán Michael Conroy says, you must look at, think about and report the confidence intervals which are likely to be very wide.

But it is what it is. If you have 54 patients, you have 54 patients. You can't wait for a decade to get more. Whatever your findings are, they will serve as stepping stones, or hypotheses for any future long-term or more likely multicentre studies.

Best of luck.

Thank you all for your answers, it helps me in writing down what I find properly. Indeed I was talking about patients, and am not performing a meta-analysis.

You are correc Lakshmi, it is what it is - and still the best available evidence!

Thanks again!

In a way, having only 54 patients is a bonus, precisely because of the phenomenon that Lakshmi Narayana Yaddanapudi mentions. Effects that show up in small samples are big effects, and likely to be relevant to informing treatment and prognosis. Effects that show up in big samples, on the other hand, may easily be too small to have any clinical significance.

This is why early-stage animal research uses such small sample sizes. They only want to pursue effects that are visible to the Mark I Eyeball test.

I agree with Ronán Michael Conroy's statement, "Effects that show up in small samples are big effects." However, they are also more likely to overestimate the true effect size in the population. See Button et al. (2013), for example. Here is their abstract (highlighting added):

"A study with low statistical power has a reduced chance of detecting a true effect, but it is less well appreciated that low power also reduces the likelihood that a statistically significant result reflects a true effect. Here, we show that the average statistical power of studies in the neurosciences is very low. The consequences of this include overestimates of effect size and low reproducibility of results. There are also ethical dimensions to this problem, as unreliable research is inefficient and wasteful. Improving reproducibility in neuroscience is a key priority and requires attention to well-established but often ignored methodological principles."

https://www.nature.com/articles/nrn3475

Article Power Failure: Why Small Sample Size Undermines the Reliabil...

Finally, the the two 2005 Lancet articles by Schulz & Grimes remain among the most thoughtful pieces I've seen on the topic of multiplicity. The second article includes a nice discussion of sub-group analysis.

Article Schulz K, Grimes DMultiplicity in randomised trials I: endpo...

Article Multiplicity in randomised trials II: Subgroup and interim analyses

HTH.

Hi all, Thanks again for your answers, but I am still a bit flustered..

When I perform a crosstabulation in SPSS, I get certain p-values.

However, I can not get confidence intervals from these, since a CI is based on a normal distribution and takes into account larger sample sizes, so how can I report this?

Are there other tests besides Chisquare of Fisher's exact test that I could perform to show confidence intervals and/or describe my results better??

Thank you.

Hi Sonja Levy. If they are 2x2 tables, add keyword RISK to the STATISTICS sub-command for CROSSTABS. The help says this causes SPSS to display a table showing the "relative risk" (with 95% CI). But IIRC, it also shows the odds ratio, in case you prefer it over the RR. (If you are a GUI user only, I'm sure you'll find a checkbox to do the same thing.)

https://www.ibm.com/support/knowledgecenter/en/SSLVMB_26.0.0/statistics_reference_project_ddita/spss/base/syn_crosstabs_statistics.html

Hi Sonja Levy. May I recommend the online statistics website openepi.com for your data?

The general guideline for subgroup analysis in any clinical study is:

I agree with the points A D Paul made in his post. But as Schulz & Grimes (2005) note (in the second article I linked to in my previous post), reviewers or editors may insist on sub-group analyses. I quite enjoyed this suggestion about how to deal with that situation, should it arise.

" The Lancet published an illustrative example.4 Aspirin displayed a strongly beneficial effect in preventing death after myocardial infarction (p0·00001, with a narrow confidence interval). The editors urged the researchers to include nearly 40 subgroup analyses.2 The investigators reluctantly agreed under the condition that they could provide a subgroup analysis of their own to illustrate their unreliability. They showed that participants born under the astrological signs Gemini or Libra had a slightly adverse effect on death from aspirin (9% increase, SD 13; not significant) whereas participants born under all other astrological signs reaped a strikingly beneficial effect (28% reduction, SD 5; p

Sonja Levy

However, I can not get confidence intervals from these, since a CI is based on a normal distribution and takes into account larger sample sizes, so how can I report this?

The answer is to stop using SPSS!

You should be using software that allows you to calculate confidence intervals for small sample sizes. You can do this (easily) in Stata or (with a little more difficulty) in R. You can also calculate very useful measures of effect size like attributable risk.

Come on, Ronán Michael Conroy! I love Stata too. But I know that SPSS can do many things that many of its users do not know how to do, especially if they are point-and-click users exclusively. (I daresay that many Stata, SAS, R, etc. do not know all the capabilities of those programs either.)

Using SPSS

DATA LIST LIST / r c n(3F5.0).

BEGIN DATA

1 1 9

1 2 45

2 1 14

2 2 40

END DATA.

WEIGHT by n.

CROSSTABS r by c /STATISTICS=RISK.

OUTPUT:

OR = .5714286 (95% CI: .2233322 to 1.4620847)

Using Stata

cci 9 45 14 40, woolf

OUTPUT:

OR = .5714286 (95% CI: .2233322 to 1.462085)

;-)

Bruce Weaver

I had no idea it was so simple in SPSS – amazing!

All those years I've wasted with Stata…

Actually, I was an SPSS user back in the seventies when it was by far the best thing going. But then came SAS…

Ha! I didn't say it was simple, Ronán. I just said it was possible. As everyone can see, the Stata code is much shorter & simpler. As a long-time SPSSer who now uses Stata too, I find that more often than not, Stata code is shorter & simpler than the corresponding SPSS code. (Ducking for cover from the flames now.) ;-)

Cheers!

Bruce

I'm going to avoid the flame throwing, and just add a paper relevant to Bruce Weaver 's point about small samples producing estimates that are often too high in magnitude (and may be in the wrong direction). The paper is: http://www.stat.columbia.edu/~gelman/research/published/retropower_final.pdf . Briefly, if you are running several tests with small samples, the ones that appear interesting (by whatever means, but suppose it is p