@Igor : How big is the deviation from the assumption? About which variances are we talking by the way? Where does the assumption of linear growth fit in the model? What is the exact model? Is time taken as a predictor variable? In case it is, why use MANOVA (as that's far from the best option for longitudinal data)? Did you check for autocorrelation?

All questions a statistician would ask you before even trying to answer your question.

Did you think about including random factors in the model? Why not using a Generalized Linear Mixed Model, or a Generalized Additive Mixed Model? These can deal more correctly with longitudinal data than MANOVA. GAMMs come the closest to the approach you suggest, but then in a statistically correct manner. They also require quite some data for a stable fit, so I won't say that these are your best option either.

So again: take everything to a statistician and let him/her take a closer look at it. Without the actual data and research questions it's impossible to even come close to a sensible advice for a "best practice".

Your question is the equivalent of:

"I have a headache instead of food poisoning, so which alternative diet would you suggest?".

Without trying to understand the nature of the headache by physical examination, not one doctor will suggest a diet (and rightfully so!). Tons of herb enthousiasts will though.

Specifically what assumptions are violated? The answer will likely depend on the assumptions themselves. For instance, if the power/sample size determination was made assuming the data to be normally distributed, and the actual data are binomially distributed, you have a major problem. But it the assumption is regarding a misestimation of variability, the method will probably remain unchanged.

Use an appropriate method. Or better: if you have to ask that question, take your data, design and power analysis to a statistician. Sounds like a no-brainer, but seriously, this question cannot be answered without a detailed look to the extent of violation, knowledge on the method and its alternatives and a solid idea about the hypothesis you want to test.

One example I see almost daily:

People often use a Wilcoxon/ Mann-Whitney test as a replacement for the Student T-test when the data isn't normally distributed. That's fine. That's not fine when dealing with sample sizes that are too small for a T-test, because the Wilxocon test is in general also asymptotic, and hence not fit for small sample sizes. For small sample sizes, you need an exact test.

Thank you, Steve and Joris! I am asking this as a general question, not related to any specific data set. But suppose there is such a hypothetical example:

Tumor volumes are to be measured repeatedly for mice in a control group and several treatment groups. For sample size estimation MANOVA is used, assuming that the tumor volumes grow linearly with time and variances are equal. But when the experiment is done, it turns out that the tumor growth curves are non-linear (in some groups they plateau, in others they can decrease over time) and the variances are not equal. Is it appropriate to analyze such data using methods other than the originally selected MANOVA? For example, develop a mathematical model which fits the data reasonably, estimate its best-fit parameter values for the different treatment groups, and compare the CIs for these values?

Thanks in advance for input from anyone interested!

The assumptions of a test do not determine the logical validity of the test. Say you wanted to do an ANOVA, but found out your residuals were not normally distributed. A nonparametric ANOVA will STILL answer the question. After all, your predictions and hypotheses to suggest them are what matter most. Remember, transformations can resolve some assumptions too, at times. Please be more specific to get a more specific answer! Good luck.

I, for one, would have no problem analyzing data of this sort as a repeated measures ANOVA, even though the sample size had been determined through a method where the correlations over time and variances were constant. It would be the best possible analysis available, and would retain as much of the original power as could be expected. However, not accounting for the heterogeneity of variance and the changes in the correlation of measures (i.e., using an incorrect specification even though that was what was used to model the power and sample size) seems naive and asking for reviewer comments.

@Igor : How big is the deviation from the assumption? About which variances are we talking by the way? Where does the assumption of linear growth fit in the model? What is the exact model? Is time taken as a predictor variable? In case it is, why use MANOVA (as that's far from the best option for longitudinal data)? Did you check for autocorrelation?

All questions a statistician would ask you before even trying to answer your question.

Did you think about including random factors in the model? Why not using a Generalized Linear Mixed Model, or a Generalized Additive Mixed Model? These can deal more correctly with longitudinal data than MANOVA. GAMMs come the closest to the approach you suggest, but then in a statistically correct manner. They also require quite some data for a stable fit, so I won't say that these are your best option either.

So again: take everything to a statistician and let him/her take a closer look at it. Without the actual data and research questions it's impossible to even come close to a sensible advice for a "best practice".

Your question is the equivalent of:

"I have a headache instead of food poisoning, so which alternative diet would you suggest?".

Without trying to understand the nature of the headache by physical examination, not one doctor will suggest a diet (and rightfully so!). Tons of herb enthousiasts will though.

Thanks again, Joris! My question is general, along these lines: if the experimental data turn out to poorly conform to original assumptions during the sample size estimation, can a different method (perhaps very different) be used to analyze the data than was originally intended?

For the hypothetical example, I chose the MANOVA approach described by Heitjan et al (Statistical Analysis of in Vivo Tumor Growth Experiments, Cancer Res 1993;53:6042-6050). Does this make sense?

Consider a nonparametric statistical alternative, where assumptions of a priori conditions are usually more relaxed, though may require some dependent variable transformation or reconstruction, which often results in a simplified, and somewhat, more primitive representation of outcome variation.

The purpose of same size estimation is to provide the best guess based on your current knowledge. It's quiet common that some of the prior knowledge are not accurate. For example, we based on standard deviation of previous studies to calculate SS but in fact the observed data in the new study have totally different SD. If we have done the experiments and found that some conditions/assumption used to calculate SS are violated, it should be ok to use an alternative but more appropriate method.

Here's a second to Fang-Yong Li...a priori power analysis is a design tool. A series of "what if"s...and frequently incorrect. Practically speaking, I suspect you wouldn't run into problems unless you attempted to report null results...and even you could conduct a post-hoc power analysis on what you were "powered" to detect with the new technique. I would only worry a little about questions of "why did they think THAT?"...but suspect you won't get many reviewers nit-picking at that level when it's unnecessary. A lot of times, the modeling in a field hasn't evolved to the point where parameters are available in the literature for the best models...and you approximate as best you can...being conservative if you can.

Subject specific: My experience has been that tumors tend to change exponentially or non-linearly. That is, the tumors grow by proliferation, which is an exponential function, base 2, right? Now, the micro-environment may itself change as the tumor grows, as probably does the tumor (hypoxia induces angiogenesis and so on)...and perhaps the tumor slows in growth as resources become practically limited. It sounds like you other conditions where you have shrinking of the tumor (yah!)...probably itself happening with diminishing returns. Both growth and shrinking may, be reasonably represented by a decay model (non-linear mixed) of some variety that permits the asymptote to be a free parameter...and would represent the final tumor size and a very powerful primary outcome...the rate parameter may not be interesting except in the case where the asymptotes are similar.

...just some thoughts!

Igor, the best way to evaluate sampling and statistical designs is by constructing a stochastic simulation (Monte Carlo simulation). The advantage of this type of analysis over traditional closed-form calculations of statistical power is that you can evaluate the performance of the design in terms of type-I and type-II error rates in the presence of assumption violations. You can than compare the performance of various sample sizes or even alternative analyses in the simulation. In this way you can test the robustness of the design. I'm not familiar with the medical literature, but these types of analysis are in the ecological and fisheries literature. A paper that summarizes the approach is....

Zurell, D., U. Berger, J.S. Cabral, F. Jeltsch, C.N. Meynard, T. Munkemuller, N. Nehrbass, J. Pagel, B. Reineking, B. Schroder, V. Grimm. 2010. The virtual ecologist approach: simulating data and observers. Oikos 119:622-635.

Good luck!

We have to put this question into context, as I think the answers are different for data coming from studies of different designs. For a clinical trial, I would suggest a re-estimation or this issue has to be discussed again incorporating the clinical information and knowledge to assess our initial understanding of the hypothesis and our chosen endpoint. If the data were from an observational study, there may be a transformation of the endpoint that help.

Igor's question is very relevant and goes way beyond the simple question of sample size; it is all about fiddling with data too much to find significant results (called data dredging)! This is unfortunately very commun in science and leads to many false positive studies to be published in high impact science journals.

Sample size is an estimation and therefore observed data are expected to deviate slightly from expectations (if this were not to be the case, studies would be useless as we could always predict results correctly). There is therefore no problem to slightly deviate analysis from expectations as long as these remain logical and are done only to fulfill statistical assumptions. Optimally, it would nevertheless be wise to plan these deviations ahead and pre-plan alternative solutions (transformation, non-parametrical analysis, etc). Finally, if we highly suspect the data will deviate, the study should be powered for non-parametrical testing.

The second point is about data dredging. This major scientific error is frequently observed in statistical analysis in biology and occurs when researchers fit the data to obtain significant results. Research then justify their methodological choices providing post-hoc made up theories on their data (reasons for ruling out outliers, justifying complex transformations related to subjects' characteristics, changing analyses method, etc.). This would be fine if conditions were pre-planned (selection criteria for mice, ruling out outliers, excluding measures under certain specific conditions, etc.) before collecting data but is a scientific flaw if it is done afterwards without mentioning it. However, never forget that if these choices are made post-hoc, p-values become senseless! The rule for p-values to make sense is "Pre-plan and execute... or explore, modify, and then validate on a new set of unexplored data".

Finally, using ANOVA to model data is like relying on an ox to plow a field. It works, but it is not as efficient as if you were using a tractor.

By pre-planning analysis and relying on adapted advanced statistical methods, I believe the important number of false positive published studies in high impact science journals could be reduced.

I could repeat what Joris, Dan and Paul said, but I won't :)

The ideal way out would be to use this new data to judge which transformation is reasonable and then to perform a new power analysis. Unfortunately, then another experiment has to be performed.

A not-so-good way out was to re-estimate the power on the transformed pilot data and use this for your conclusions.

You cannot change the data collection. You can modify the distribution by various methods. So remember one purpose of the Power estimate is to plan to optimise resources and estimate probabilities of inferential errors (I & II). These have to be reasonable to progress. If you detect a difference it had sufficeint power if not it was under powered.. but the most important thing is the apriori hyopthesis and the 'clinical' significant of the effect size estimate. p values define probability not the 'so what?' part of the question (especially in health). ( I do not believe in post hoc power analysis for the above reasons).

Yes, post-hoc power analysis is essetially as bad as a post-hoc decision about the distribution and the transformation and test that is applied. Whereas a post-hoc decistion about transformations and tests still give some useful results (at least the estimates, given a careful interpretation(!) also their precision estimates like confidence intervals), the meaning of the p-values is lost. So any reliance on p-values or fixed significance criteria is sensless. And because a power analysis is just done for this purpose, a post-hoc power analysis is completely pointless.

Interestingly, this sheds some light on an important but vastly ignored problem: All power-analyses and tests assume that the model and the error-model (i.e. the shape of the distribution) is *correct*. We know, a priori, that all models are wrong anyway. So here is the question *how* wrong they may be to still give "good-enough" results and conclustions. Now, what is "good-enough"? These questions are never addressed, and there are no statistical means to answer such questions.

I think Paul stated the "purist" view quite nicely, but I don't see things as black and white as that. I see them more situational and nuanced. True, p-values become less than fully accurate when post hoc deviates a priori, but if my p-value is .0001 and I haven't gotten into serious rummaging around and cherry picking then I will simply note the post hoc decisions made and still claim significance. I think that position would make it past most statistical reviewers. Actual practice is often far worse than that. I believe it is in the perview of a statistician evaluate the risk of false discovery both formally and informally. Granted, some will try to abuse the informal side, but that is what disclosure of rationale is all about.

David, I agree. But you only talk about a "significant" finding. A power analysis is not really required when you are looking only for significant findings. The problem are non-significant findings. Given you have *not* assured an acceptably high power (say 90%) but decided on alpha=0.05 and you get p=0.06, then you simply have to keep your mouth shut. Otherwise it would be like a judge in a court believing the words of a supposed attestor who is not known and where it is not even clear whether or not he has had only the possibility to witness the crime under investigation. On the other hand, given the power is assured to be 90%, you can conclude with 90% confidence that the "true" effect is smaller than the minimum relevant effect.

My point was a little bit stronger, also including conclusions about "significant" results. a post-hoc decision on a data-transformation invalidates the p-value, and the rate of type-I errors is not anymor controlled as expected. Badly, the empirical rate is greater than desired. However, I think the effect is not verly large. I tried an example simulation with gamma(1,1)-distributed data, 2 groups, n=5 values per group. The empirical false-positive rate assuming normal distribution, estimated by 10000 simulations, was 0.040, so a little lower, or better, than 0.05. Always using the Box-Cox transformation to get normally distributed residuals gives a type-I error rate of 0.056, what is, as I have to admit, not really much higher than the expected 0.05, but it is higher.

Here is the R code:

require(MASS)

bc

Jochen - great idea to use a quick simulation to scope out the extent of imprecision in alpha values for sample cases - nothing better than checking with simulations from time to time to ensure that a statisticians statistical judgement is well grounded!