Zsolt, I never said that multiple testing must be avoided. There are often research questions that would be answered better by a specific test - but many researchers do not know how to do this and they instead do multiple comparisons between whatever groups they find and run into a multiple-testing problem that is even not adequate to tackle the actually relevant research question... Also many tests are unneccesary or at least irrelevant to keep an error-rate (e.g. comparisons of several control-treatments to a baseline, that are often all fomally tested [for whatever reason]; at least such tests are not addressing the main research question, they are at best supplementary. A multiple testing correction including such tests seems non-sensical).

But multiple testing can be sensible or required. It depends on the research questions and how the experiment was designed. However, one needs to think if a control of an error-rate is required or sensible, or if the tests are performed only to get the top-n "most significant" candidates for further research.

2nd Point:

As long as you somehow select the data-processing or test procedure based on features of the data, you introduce a bias in the further statistical analysis. It does not matter if this selection is based on a formal hypothesis test or on the "informal" inspection of the data.

It is typical in research that we get data where we do not understand the whole data generating process. We need to explore the data, we need to find out its properties, its distribution etc. This is explorativ. The aquired knowledge could be used to generate or to modify particular hypotheses. If the hypothesis is to be tested in order to control an error-rate, then the same data must not be used again.

You can understand this when you understand that a hypothesis test (as well as the construction of a confidence interval) is a procedure, and the error rate (or confidence level) is a property of the procedure (and not interpretable for a single case). These are frequentist tools! So it is not about a single, particular test - it is about the whole procedure from data to the p-value (or confidence interval).

If the decision which test to chose (based on the current data) is thus a part of the procedure, and this destrys the independence between data and test. The procedure will not work as expected, that is, rejecting the null whenever p

Wilcox (2005, p. 377, Introduction to Robust Estimation and Hypothesis Testing 2nd ed) discusses a method by Munzel and Brunner (2000 Biometrical Journal) and a method by Choi-Marden (1997, JASA). The later he calls "a multivariate analog of the Kruskal-Wallis. They are both multivariate methods when you have multiple independent groups. So I think these might be the sort of thing you are looking for. His functions are called mulrank and cmanova. These are written in R and to access his functions see http://dornsife.usc.edu/labs/rwilcox/software/.

Alternatively, the tsplit function in the R package WRS2 (which is based on Wilcox's work, but intended to be more user-friendly) is a "A robust two-way mixed ANOVA for trimmed means", which also seems like it might be what you are after.

I am sure these is more out there (and probably Wilcox has done more recent things too ... he is one of the clearer statistics writers), but this should get the discussion started.

"It is not rare in behavioral tests that the data is not normally distributed, thus, we need to use non-parametric tests" 

I don't subscribe to this. There are many other parametric probability models available, and some of them may turn out to be utmost useful in particular cases.

Further, "standard" tests are about expectations, and rank-based tests are about distributions. These are different things. So if testing expectations is your aim, you can not simply jump to "non-parametric" tests just because the data is non-normal. At this time some deeper thinking about the response variable and the kind of data should start.

"Escaping" into non-parametric testing is often a clear sign that the researcher has not understood the data (well enough) and that he is not really clear about the analysis (and, thus, about the relevant research question). Further, it usually shows that the researcher is not really working on substantial science and rather tries to rely on some p-values he fiddeled together.

Thanks for the answers.

Thanks, Daniel, for the suggested papers, I will look up them.

Jochen, you telled that I am wrong, but you did not suggest any solution for this problem. I know that there are fundamental differences between the non-parametric and parametric tests, but Kruskal-Wallis and Friedman tests are often referred in basic statistics books and in some statistical softwares as alternatives of ANOVA, if data is non-normal. Thus, I don't think that I am the only, who used them in this - maybe wrong - way. They are often used for hypothesis testing about expectations in publications, too. (However, I know that everything can happen in publications...)

So, I am really interested in methods, which can be used as alternatives of ANOVA, if the assumptions are violated (and transformations did not help).

I agree with Jochen's advice regarding ranked-based and non-parametric analyses, and don't use them often, but do not feel as strongly as him.  You should not jump immediately into the ranked-based methods as soon as you see some assumption of the common ANOVA (or whatever) not meet because you lose something. You lose estimate these parameters, and often your theory should be about the parameters. If you can specify a model that accounts for things like the shape of the distributions then you will gain more from your analysis.

It is worth adding that the very distinct phrases non-parametric, distribution-free, and robust often get confused. If the reason that you want to use ranked-based tests is because of outliers having too great an impact on your estimates, a worthwhile alternative is using different loss functions which are more robust.

Also, for a list of relevant R packages see:

http://cran.r-project.org/web/views/Robust.html

Dear Zslot,

I did not say that you are wrong. The way you describe is a common "misbehaviour" in the statistical analysis, and this is - as you said - largely due to stupid things written in so many textbooks and propagated by so many software(manuals).

What else can you do?

First you need to try understanding what the data really is.This can be a very hard job and there is no cookbook recipe to help you. Some simple answers may be: proportions, counts, or concentrations. But your particular case may be much more difficult. I consider this really scientific work to answer the question what data one has. One needs to have a clear and deep understanding of the whole experiment and its scientific background to get an answer.

The next step is to identify -or even develop!- and appropriate probability model to express what one can expect from such data. Again, for the simple answers there are the usual suspects, like the beta for proportions, the Poisson or negative-binomial for counts, the gamma for concentrations, and, well, the normal for "measurements with symmetric errors". This may also include thinking about transformations. For instance, at a basic level of "uncertainty" effects may multiply into the final read-out. In such a case a multiplicative model (= using the log transformed response) may be appropriate).

And finally the (functional) model must be approriate. All relevant predictors / inflencing variables / covariates must be correctly considered, also possible relevant interactions, and the relation between the "linear predictor" and the response must be linear, what may indicate the transformation of some predictors. This is another very difficult task.

As a final note:

If you look a given set of data and then decide about the model and the analysis, then you can not well interpret p-values (unless they are really really tiny, what still may be interpreted as some evidence against the corresponding null hypothesis)! If you want to interpret p-values, then you have to do the tests on data that was NOT USED to chose the model. This also applied to this "parametric -> non-parametric test jumping": if you look at the data and then you decide whether you do a parametric or a non-paramteric test distorts the meaning of the p-value. You actually have to get the p-values from a different set of data (or at least multiply them by 2)!

* this is associated with the multiple-testing procedure; statistically looking at the data means that you consider both tests and then decide for the one that likely performes better on *this* particular data; so actually the p-value should be corrected for multiple testing

I would make sure you understand the assumptions of the model you are using.  In anova, the assumption isn't that the _data_ are normally distributed, but that errors (residuals) are normally distributed. 

(I apologize if that's not what you meant; normal "data" in the question isn't entirely clear.)

You might also consider transforming the response data.   Some things in the world just follow different distributions.  In my work, pollutant concentrations in water are often log-normally distributed.  It's no trouble to transform into a normal-ish distribution, and use parametric tests.  You just have to be cautious with how you report your results.

For anova assumptions, see for example:

http://www2.hawaii.edu/~halina/603/Transform.pdf

Most nonparametric tests based on ranks (like Kruskal-Wallis and Friedman) do not extend well to additional factors. The reason for this is that the assumption that all rankings are equally likely under the null hypothesis does not work when testing the significance of one factor when another factor might have variation in the outcome measure.

Any extension of these nonparametric tests would therefore no longer be nonparametric. That doesn't mean that you can't use them, but it means that most of the motivation for choosing them disappears.

There is an alternative, however. Ordinal logistic regression is able to handle ordinal data like ranks and it is invariant to any montone transformation of your outcome variable. And ordinal logistic regression extends easily to multiple factors. It does have a pretty strong assumption of parallelism, but this is something that you can easily check. It certainly would be less restrictive than the assumption of normality.

Great suggestion from Stephen. Give ordinal regression model a try and see how it works out. That has been my choice when handling data similar to yours. Daniel is correct on the use of parametric or non-parametric tests, however there is no invariability in the use of these tests. STATISTICS IS AN ASS - JUST LIKE LAW.

Glad to contribute.

Thanks for all the answers.

Jochen, I strongly agree, that multiple testing must be avoided, and thanks for the detailed explanation of the issue. Do you think, that if I test the distribution before (without any hypothesis tests) and choose the model on the basis of the distribution (or other descriptive statistics), can also affect the reliability of the result?

Another issue is the type of variable: in some cases, it is quite obvious, which type of tests should be used (e.g., binomial variable, counts). In our experiments, we also use different types of variable, e.g., ratios, percentages, rates (pl. count/sec), time (continuous) or variables which can take on only whole number values in a range. Some of these cases are really more difficult.

However, occasionally we have variables that (in my opinion) could follow normal distribution, but in the certain experiment it didn't. What do you think: are rank tests appropriate alternatives of parametric tests in these cases, or is it better to use the previously chosen parametric model (e.g. an ANOVA) even if some assumptions are not met?

Salvatore, You are right, I mistakenly referred to the distribution of the variable, not to the distribution of residuals. Again a bad habit from bad textbooks :). Thanks for the guide.

Stephen, thanks for the explanation about why rank-tests can not handle multiple factors. Thank you for the suggestion of ordinal logistic regression. So, if I transform the data into ranks, than can I analyze them with the ordinal regression?

On the whole, choosing the appropriate statistical methods for biological studies is not a trivial task... I think, the main problem is, that most biologists (like me :)) haven't got enough knowledge in statistics to develop specific statistical models for their needs, therefore textbooks offer "instant recipes" for general purposes. If you look at publications, it is common, that ANOVA and t-test are used for almost everything. Maybe, the motivation behind simple models could also be the doubt, that even the reviewer will not understand a more sophisticated model...

"Maybe, the motivation behind simple models could also be the doubt, that even the reviewer will not understand a more sophisticated model..."

Or even worse: if you are using an appropriate statistical model, but which is not very common in your field (as you said, most of the time your peers use ANOVA and t-test), some reviewers might get suspicious. "Why did he choose this fancy statistic? Probably, he searched for a test which makes the results significant, because the standard test doesn't"

This is the problem, if your peer-reviewers used to learn their statistics stuff with the same bad textbooks as you did.