How many treatment and how many control groups are there?

I have one control and 5 treatment groups 

What is your sample size, that is number of individuals per group? Does three experiments mean you have n=6x3=18 total and of that 3 are control? Is your response variable binary (ie dead or alive cells)?

Cell viability (response variable) is expressed in percentage (quantitative continuous variable).

 After a certain time of drug exposure I measure cell viability in 1 control and 5 concentration groups. I repeat the experiment independently 3 times. The results from 3 experiments look like this for example:

Drug concentration,mM      Cell viability, %

0                                          100;100;100 

0.25                                       95.5 ;  99;  92

1                                           80;   82,6;     81

2                                            72;  73;  68.9

5                                          45;   46;    39,9

10                                         17;  22;     20,1

These are the results for only 1 cell line, one treatment duration. I want to check, if drug concentration significantly affects cell viability.

Use ANOVA or Kruskal Wallis if this is not a dose-finding study.

It looks like you will be able to use ANOVA for your data.

To test the assumptions of anova, you want to plot the residuals from the whole model as a histogram or qq plot. The residuals should be fairly normally distributed. You also want to plot the residuals vs. the predicted values (or something similar) to check for homoscedasticity.

Ideally, you could combine the data from the three (?) experiments into one model. It's not clear to me from your description what variables you would need to add (Cell line ?, Duration ?, etc.), but you may want to consider this. Look up "general linear model".

If you are not comfortable with your plots of residuals, you can use Kruskal-Wallis test.

* * *

That being said, there are probably better ways to analyze these data.

On obvious one is to treat Concentration as a continuous variable, and use linear, polynomial, or nonlinear regression to relate Viability to Concentration.

If the data are such that you have the counts of alive cells and dead cells for each observation, logistic regression would be a better approach. In this case, Concentration could be treated as either a continuous or a nominal variable.

Finally, with proportion data, beta regression is appropriate. Also in this case, Concentration could be treated as either a continuous or a nominal variable.

* * *

If this kind of data in your field is always treated with anova, then you are probably wise to follow this convention.

But if that is the case, I could see a publication or conference paper explaining the different ways the data could be analyzed, and the advantages and disadvantages of each.

The assumption of "normal distribution" requires that your sample(s) was/were drawn from a "normally distributed" population. It does not mean that the sample itself must be "normally distributed". Therefore your statement "...and this small sample cannot have normal distribution" is not appropriate and makes no sense.

Regarding the purpose of your comparison - "I want to check, if drug concentration significantly affects cell viability" I would definitely follow an approach where drug concentration is treated as a continuous, independent variable, namely regression (linear or non-linear). My decision is mainly based on the fact that you know a priori what are the possible outcomes when you increase the drug concentration: (a) no change in the cell viability or (b) a monotonic decrease in the cell viability. ANOVA or t-tests don't take into account the drug concentration (in this case, only the cell viability) and therefore they are "missing" a really important piece of information.

But, as Salvatore stressed, there are advantages and disadvantages in using different approaches.

Thank you very much for the replies! They are very helpful

 Dear Aisté, Kindly note that it is common practice to recommend useful and valid answers and to "report" misleading/erroneous ones. Recommending is not only a way of ranking the contributions but also the contributors. Regards, Luis

Luis, just wanted to mention that one can perform trend analysis with ANOVA to say that cell viability has some trend (linear, quadratic etc). If this is the very purpose even Aiste can use it for her data to find a trend in cell viability against different drug concentrations.

I totally agree that other methods than ANOVA are more appropriate if drug concentration is needed to be treated as continuous.  But according to her description, she just wants to find an association. So a simple ANOVA or KW seem to suffice.

@Fausto, You're right about using all the data (instead of the averages) and that the reported data is suitably described by a quadratic model.

The "right" model is however a log-sigmoid (the dose-response curves, in pharmacology), but the limited concentration range (upper end) turns the quadratic model "as good" as the sigmoid to describe the reported observations. Just for curiosity, you may try to fit  Y=100/(1+10^(-log(C/IC50)*Slope)), where IC50=3.80 is the concentration that inhibits 50% (the drug potency) and "Slope"=-1.26 is how steep the transition from 100% to 0% is. 

Regards, Luis

@Fausto,

Yes Y is the viability and C is the concentration.

Please note that Viability is NOT transformed in any way, only the drug concentration is log transformed. Therefore if the Y residuals are normally distributed (within each concentration) they still will be after the log C transform. I didn't check numerically, but it can be seen that the quadratic model describes better the "2" and "10" drug conc  and therefore the residuals of those concentrations may be slightly biased for the sigmoid fit. If that's the case, the problem is not the model (as it is the standard physiologically based model used in pharmacology for ages) but due to experimental error.

It does not matter if the quadratic model fits well, or even "better" than the sigmoid (for a limited range of drug concentration), if it has no physiological/pharmacological meaning. Take a look at the accompanying  chart with both models fitted. The zero drug concentration was changed to "0.001" to allow the log transform. 

Regards, Luis

@Fausto, Hi, I agree with you that the C log-transform does not changes the ANOVA outcome. Thanks for the exchange of comments and I guess Aistè already has some extra thoughts on the subject! Best regards, Luis

Good contributions here.