Recently I have come across several research publications that present multivariate regression analysis with small N's. The manuscripts throw around comments about thousands of surgical patients and hundreds with this or that condition but the actual studies have N= 36, 64 and 78. I am aware of the "rule of thumb" of [N= 100 + (number of variables)] in lieu of a-priori power analysis. None of these studies presented a power analysis (a-priori or post-hoc) but one commented that the study "might suffer from lack of power".
Why do reviewers and editors propagate the use of poor statistics? Should "WE" not demand more from our researchers? I believe that presenting these studies "as is" without multivariate analysis would more beneficial to the reader, the poor statistics may mislead the reader. (or is that the intent?)
Thank you for considering my frustrated rant.
This is my follow-up to Patrick's and Ronán's posts. Both made a good points, and I think these must be seen in conjunction with my statement. Taken together (in the order of appearance in this thread):
- small-N studies are not per se problematic. They are statistically valid (given they are done correctly and their underlying assumptions are reasonable).[Jochen]
- small-N studies can be done many times on the same or similar questions with little effort, and only "significant" findings are reported. The resulting publication bias does not consider the multiple-testing problem, so that the significance gets over-estimated. [Patrick]
- small-N studies are sometimes neccesary as there is no large sample existing or accessible. [Ronán]
Ronán's point "compensates" Patrick's point, because rare things won't be analyzed very often. But the reader has to know and to consider the rarity of the studied object when interpreting the significance (what was addressed by my post, without being very concrete on that point). Given a p=0.03 for an N=6 study on a rare disease has a different meaning to me than on a study of the impact on eating eggs on the blood cholesterole level. An N=10000 study with p
Exactly, It's a valid point.
We should not enamoured of multivariate statistical techniques until we are completely aware of all the assumptions.
I have seen many real examples especially from medical research(across globe), researchers are using classical statistical techniques without knowing the assumptions, without knowing the validity of their findings. They have been using techniques irrespective of the sample size requirements, the distribution assumptions etc...
They must understand the prerequisites otherwise their results will mislead and will not be statistically valid.
Would recommend to go with Bayesian, when the N falls far short of "acceptable" minimums.
Regards,
Abhay
The problem would not be that severe if "WE" would stop thinking that the results of a statistical analysis of some set of collected data represents some general "truth" or that it would be sufficient to simply have p
Jochen,
Thank you for your response. I believe we agree. My issue is the use of a statistical test when the N falls far short of "acceptable" minimums. My comment about power analysis was to illustrate that there wasn't an attempt to address the shortfall in N by addressing effect size. Furthermore, what is wrong with presenting one's findings without attempting to represent a study as if it is of greater magnitude than it actually is? I find that often, readers of the literature grasp on to these small studies (with inappropriate statistics) and present it as "evidence" for changes in practice. The effort (money) wasted could be used with greater effect.
Rgards,
Christopher
It is certainly fraud to report larger sample sizes than actually used.
But I think that small sample sizes are not the key problem. I don't think that statistical tests are invalid because of small N. Well, maybe we have to be more specific about the test philosophy: Fishers significance tests aim to reject the null hypothesis. There is no concern of power because there are no substantial alternatives, and failing to reject the null only means that the data is (for whatever reason) inconclusive w.r.t. the null. This is a bit different for Neyman/Pearson hypothesis tests where two substantial alternatives are formulated and the test aims to decide for one of these alternatives. Here we do need the concept of power (beta) and size (alpha) of the test to allow a rational decision criterion. Selecting the alternatives as well as alpha and beta requires a lot of a priori knolwedge and a loss function. Since all thise things are usually unknown in basic research, hypothesis testing is not really an option (it usually remains completely unclear if the selected alpha and beta ensure a "rational" decision!). All this is terribly confused in literature where people talk about "null hypothesis significance testing" where these actually incompatible philosophies are mixed into one illogical conglomerate.
I think that factually most of the tests reported in papers are essentially significance tests (Fisher), where the only aim is the rejection of a null. There is no lower limit of an "acceptable" N for such tests. Too low Ns will lead to many "failures to reject the null". This can be an economic problem (I do agree here!) but that's not a scientific problem (given the correct interpretation). However, when even a small N experiment gives data that is very unlikely under the null does make the null suspect (-> "evidence against the null").
I see a bigger problem in very large studies where negligible violoations of the assumptions (what includes that the unrestricted statistical model is correct) can cause "highly significant" findings. In such cases, to my opinion, it makes no sense to look for the significance at all. Instead one must really and specifically interpret the estimate directly, and this interpretation must consider subject knowledge and not only the mere numbers.
Jochen,
Again, thank you for your well thought out responses and involvement in this thread. I agree!
Specifically, when a sample size is small and multiple regression analysis is performed (especially with continuous variables) larger differences can be identified where none truly exists. Do to the availability of computer statistical programs, researchers will apply multivariate analysis without understanding the assumptions related to linearity, normality, homogeneity of variance-covariance, binomial and continuous variables.
The analogy that I have often used to explain multivariate analysis to students and colleagues begins with and example of 100 subjects (N) and 20 rooms labeled by height (x). The subjects are placed in their respected rooms, then seating areas are divided within the rooms representing other variables (a,b,c,d,e...) of interest. as each variable is held "constant" there is a point where a seating area will not contain more than one subject (or no subject at all). The smaller the N the sooner this occurs. Even if the study population is normally distributed, if two subjects fall two SD apart (within the normal distribution), the reported results are significant and misleading to the causal reader.
I have often recommended to authors that their manuscript should be presented as a Pilot Study when they have N's smaller than reference literature. Perhaps when multivariate analysis is utilized in similar studies, the authors should be encouraged to represent the results as "We don't know but... and we are now exploring...."
Although, I still believe that once one exceeds bivariate analysis the N must be large enough to support the analysis. Do we not plan out our studies prior to implementing? Are Institutional Review Boards (IRB) requiring Methods sections that include the planned statistical analysis, prior to approval? If the answer is YES to both then why does the phenomenon continue?
Thank you for engaging in this discussion.
Blessings,
Christopher
I just want to mention that most statisticians use the terms multivariate and univariate to describe the number of outcome (dependent) variables in a model, and multivariable and univariable to describe the number of explanatory variables. Unfortunately, these conventions are often not followed in the medical (and epidemiological & health-related) literature: Authors often say multivariate regression when their model has only one outcome variable.
HTH.
https://www.ncbi.nlm.nih.gov/pmc/articles/PMC3518362/
Bruce,
Great point. Thanks for the reference.
Regards,
Christopher
Jochen, while I generally agree, I think the point about small-N studies not being harmful (agreed in concept) needs to be taken in tandem with issues in the "sociology of science," where the results we see in print are often not representative of the results obtained in the broader population of studies. Publication systematically biased toward p
Speaking as a person who has to deal with small N studies, I'd like to remind people that there is not an infinite population of motor neuron disease, resistant ovary syndrome, psychogenic non-epilepsy seizures etc. At the risk of overfitting (and a lot of thought has to go into the modelling as a result) we have to make bricks with whatever little clay we have.
This is why there's a lot of systematic reviews in the health sciences!
Hello all,
First, thanks Chris for placing this topic in play; it is easy to see that many folks have strong and well-considered opinions, and the information presented is certainly helpful to those doing research. Dr. Conroy's comment is especially poignant; those in the orphan disease / orphan drug world by definition will have small n's. Reading everyone's comments leads me to believe that the n is not the problem, the over-generalization of the findings is the problem.
I frequently refer my students to a quote from Evers and Maze's Pharmacology text, in which they point out "If we knew quantitatively all of the factors affecting the distribution, elimination, and sensitivity to a drug in an individual patient, we could predict the time course of a drug exactly. However, we know only a minority of all the aspects of the dose-response relationship and, unfortunately, an understanding of one important factor generally does not explain the bigger picture.” (2004, page 3).
The over-generalization of findings is similar to this. Someone performs an experiment, finds one group better than another, and concludes that all humans everywhere should have this thing. Naturally, this does not include research done well and with forethought. While many phenomena are quite complex, we cannot use a small sample to look at each variable lending itself to that complexity. I believe finding a significant difference, and then accurately demonstrating the nature of the group where the difference was found (which allows the data to speak for itself) is far more germane to scientific advancement.
The comments here are greatly appreciated, and thanks Chris, for starting this very interesting string.
Cheers!
Rich
Ronán, certainly a fair point. I personally fall as a subject in a similar category. In that situation, I think the way to go is what I implied in my previous answer. Report every study and every test, then, as you say, systematic review. It's the small-N combined with publication bias that creates what I see as the major problem. Even with publication bias, this is where open science registries can play a role, so that the meta-analyst can find more of the non-significant results.
Pat
I've been reading this post with a great deal of interest and I appreciate the input and insight from some very highly accomplished scientists. For what it is worth, my philosophy is that prior biological knowledge should inform your statistics and not the other way around. This isn't a particularly unique philosophy, but it also doesn't seem to be very commonly held. Fellow researchers will often abandon valuable data or misinterpret the biological significance because a statistical test "told them to", or a reviewer for a journal has a strong fixation with seeing statistical significance, all the while missing the biological significance. In this regard I have to generally agree with Jochen and Ronan, but everyone else has equally valid points.
My apologies Chris, I know this doesn't speak to the specific question you posted (never mind that I would not have anything valuable to offer), but I thought I'd chime in as the conversation seemed to be heading into a discussion of the general use of statistics in science.
This is my follow-up to Patrick's and Ronán's posts. Both made a good points, and I think these must be seen in conjunction with my statement. Taken together (in the order of appearance in this thread):
- small-N studies are not per se problematic. They are statistically valid (given they are done correctly and their underlying assumptions are reasonable).[Jochen]
- small-N studies can be done many times on the same or similar questions with little effort, and only "significant" findings are reported. The resulting publication bias does not consider the multiple-testing problem, so that the significance gets over-estimated. [Patrick]
- small-N studies are sometimes neccesary as there is no large sample existing or accessible. [Ronán]
Ronán's point "compensates" Patrick's point, because rare things won't be analyzed very often. But the reader has to know and to consider the rarity of the studied object when interpreting the significance (what was addressed by my post, without being very concrete on that point). Given a p=0.03 for an N=6 study on a rare disease has a different meaning to me than on a study of the impact on eating eggs on the blood cholesterole level. An N=10000 study with p
Jochen, agreed on all counts.
Sadly, the field I was trained in was social psychology... Not such good modeled behavior for researchers! We both mentioned registries, but they still require a great deal of possibly-painful honesty from the researchers, as you indicate by your Western-blot example.
I'm a strong advocate of registries, but I'm not a PI and I haven't consistently been able to push that POV at my institutions. And for some types of studies, the registration process can be a fair bit of work. Doesn't keep me from trying.
- Badges
- Science topic
- Similar topics
- Mathematics
- Statistics