Jochen,

[...] this is exactly what I tried to say:

[...] to argue that the assumptions should be reasonable, driven by theoretical considerations, and consistent with the data at hand. In principle, proper consideration of all theoretical considerations will yield an acceptable analysis. But consistency with the data seems to be a useful safeguard against omitting relevant details.

Any reasonable person could understand what you meant to say. But, I'm not necessarily a reasonable person and it wasn't exactly what you said. :)

This I do not see:

Also, he is more generous concerning the validity of the assumptions than I am.

I think we share pretty much an identical point-of-view here.

My implicit claim here is that in practice one very rarely encounters the case where the assumptions leading to normally distributed errors are reasonable, consistent with a strong theoretical analysis, and consistent with the data. It's a more pessimistic version of the old adage, "All models are wrong, but some models are useful." As a neuroscientist, I might be too close to the (relatively) recent crisis in the lack of reproducibility in biological research to make an unbiased assessment of the quality the statistical analysis in the average paper; my bayesian prior is that the model is wrong.

In such cases the approximate solution to the right problem (provided by non-parametric methods) is often more useful than the exact solution to the wrong problem (provided by parametric models).

I was trying to be too clever here. It's a rough paraphrasing of a quote by Tukey. In the proper context, it seems to me that he was more or less correct. It was my fault for not providing sufficient context. You are absolutely correct that simply being a non-parametric test is not enough to make a method correct. One still needs to choose an appropriate non-parametric method. And even in that case, there is a convenience fee: when the assumptions are correct the parametric models almost surely provide more accurate estimates. In other words, if the assumptions are reliable a non-parametric test discards useful information. However, if the assumptions are not reliable, then the increase in precision is merely an illusion.

Incidentally, your example is an illustration of why I often recommend bootstrap resampling: it (usually) addresses the intended question and is almost idiot-proof. Bootstrap resampling would be almost magical if not for the associated "convenience fee" of computational complexity and discarding useful information provided by well founded assumptions.

if they are the same animals in 3 different days.

It may be a repeated measures analysis can help. Although the effect of the product of sample size is important.

Luck and Succes

A simple permutation test might do the trick. Here's an explanation and illustration with R.

cheers,

Shane

set.seed(100)

# generate two samples with differing means

b

What do you want to test for? 

If you want to test the effect of "STRAIN" and:

1) You used different subjects on each day, you have a 2-way ANOVA. One of the factors you test is "DAY", the other is "STRAIN". 

2) You use the same subjects each day, then you have the repeated measure suggested above. 

If you want to test sources of variability, that is a different problem. 

Thanks for the answers! I have different animals on the 3 days... I'll look into your suggestions!

Srefan - 

I know that hypothesis testing is a long-held standard, and I used to work in that area and developed a test long ago, but the problem with any hypothesis test is interpretation.  In particular, a p-value is a function of sample size.  Change your sample size and you can change your conclusion.  That just isn't logical.  (Please see the link below.)  So, I think that confidence intervals tell you a lot more useful, practical information.  

Enjoy your research. 

Thanks  - 

Jim 

Article Practical Interpretation of Hypothesis Tests - letter to the...

I'm not sure if the methods suggested apply... to be more precice here an example:

Day 1: Mean Strain A = 1, Mean strain B = 2 (n=4)

Day2: Mean Strain  A = 7, Mean strain B = 9 (n=3)

Day3: Mean Strain A = 3, Mean strain B = 4 (n =4)

On all 3 days A is smaller than B. How to calculate a p-value?

Could I normalize to strain A that the mean is always 1 and then do a ttest?

Or should I make 3 ttests and combine the pvalues with stouffer or fischer?

Is there a more elegant waz?

Thanks for all your help!

Stefan! if the main Hypothesis was to compare A with B through the different time, this is a case of repeated measure, but the sample size is too small, maybe the source of noisy data. T-test dose not work (not only due to low sample size, but due to type of hypothesis). I suggest using transformer function (for example Ln), and then employ repeated measure (if the data show normalize).  

Thanks for your response! It's not a repeated measurement. New animals every day. I want to see if A performs better than B, and combine all data.

You can look at several of my papers on strain differences. If your n=4 you should use nonparametric methods.

non parametric test..signed rank sum test.

i think will be proper as due to the small n you can't assume normality in data and proceed with parametric tests using t statistic.

What is your response variable? 

Since you have 2 factors to test, DAY and STRAIN, a 2-way ANOVA or a GLM will probably do what you want. DAY will be a random block effect. 

Depending upon your response variable, that will determine the proper method for analysis. In any case, you want to test both DAY and STRAIN. 

You cannot use Sign test because the minimum number required of n is n=6.

I suggest Kruskal-Wallis test. If you write T-test then I think of Sign test, if you write t-test then I think of Student t-test.

Stefan,

Are you allowed to post the raw data so we can see it? 

It could be that the underlying data distribution is skewed, and t-tests are sensitive to skewness. You may try taking logarithms of the data, and then performing the t-test on the logs of the values.

Have you solved it already?

Andrew gave the correct answer. Use a two-factorial model that accounts for day-to-day variability. This is simple and straight-forward.

Here is a work-around you can apply when you are unconfortable with a two-factorial model (example calculations based on the values you gave):

Calculate the mean values per day

Day 1: M1 = 1.5

Day2: M2 = 8

Day3: M3 = 3.5

For each Day, subtract these mean values from the individual measurements of the corresponding day:

Day 1: x1,1* = x1,1-M1;   x1,2* = x1,2-M1;  ...  x1,4* = x1,4-M1; ...  

Day 2: x2,1* = x2,1-M2;   x2,2* = x2,2-M2;   x2,3* = x2,3-M2

Day 3: x3,1* = x3,1-M3; ...

These data are now "adjusted for day". You can subject these adjusted values to your t-test.

Jochen how do you consider n=4 normally distributed?

Dear Béatrice, normal distribution is an assumption. It may be reasonable or not. Stefan talked about a t-test (his only problem was a lack of power due to day-to-day variances). So he made this assumption. He should have thought about it. Questioning the assumption was not part of the problem he described. There is also far too little information given to effectively discuss the reasonability of this assumption.  

To check if this assumption might be really unreasonable one might have a look at the present data. And here it is important to see that the assumption is not that the data in each group must be normally distributed. The assumption is that the errors (=residuals) are normally distributed (in the whole model). So there is (in the given example!) a total of 4+3+4 = 11 values = 11 residuals. This a bit more that just n=4, but still quite little for sure. Not much data = not much information = no reasonable way to reliably check the assumptions. Nevertheless one could give it a try and make a normal-quantile quantile plot of the residuals and see how it looks like.

But again: this is an assumption. Note the word "assumption". It is not a "requirement". Surely the assumption must be reasonable, otherwise the results are highly questionable. And reasonability can best be argued by theoretical considerations, rarely by looking at the present data. I know that this is quite different to the common practice, but it is much more in-line with common sense (and with logic, and -by the way- with test theory).

PS: There is another assumption: independence (of the errors! again). Obviousely, Stefan found that this assumption is not reasonable for his data. The errors seem to depend on the day. By subtracting the "mean day value", this dependency is removed, and these residuals are then independent of "day". There might be a hell lot of other dependencies, but we can not know anything about it. We can only assume that there are no more (relevant!) dependencies.

[Edit: added a picture of QQ-plots of 64 random samples for size 11 from a normal distribution (left panel) and a log-normal distribution (right panel). 11 values can be enough to identify a log-normal distribution, but it is not always clear; also, not all QQ plots of really normal distributed data looks as if the assumption was justified]

The main idea is though to test for mouse strain differences so why not simply compare all values from strain A with all values from strain B?

Jochen is quite correct, in my experiences here, he usually is. Although I would tend to argue that the assumptions should be reasonable, driven by theoretical considerations, and consistent with the data at hand. In principle, proper consideration of all theoretical considerations will yield an acceptable analysis. But consistency with the data seems to be a useful safeguard against omitting relevant details. Also, he is more generous concerning the validity of the assumptions than I am. 

One way in which Stefan suggested he might approach the problem was the following:

Could I normalize to strain A that the mean is always 1 and then do a ttest?

This approach seems inconsistent with the hypothesis that the errors would be normally distributed after subtracting the mean value per day. My suggestion was based on perhaps a too pessimistic assumption that nothing is known about the population distributions. Small sample size aside, in such cases the approximate solution to the right problem (provided by non-parametric methods) is often more useful than the exact solution to the wrong problem (provided by parametric models).

There is one more point I would like to make. We have, up to this point in this conversation, seen several fundamentally incompatible approaches to analyzing your data. I think that much of the variability can be attributed to the fact that we don't really know much about your data and how it was acquired or any theoretical basis derived thereby to explain the variation in your data. As Jochen states, these details will largely dictate which approach is optimal.

Also, although it may occasionally be impractical, best practices would suggest that such details of the analysis should be decided based on theoretical considerations before the data is analyzed. Now that you have already started to analyze your data, it will be much more difficult to construct an analysis that is not biased by a preconception of what the correct answer should be. 

Shane, this is exactly what I tried to say:

[...] to argue that the assumptions should be reasonable, driven by theoretical considerations, and consistent with the data at hand. In principle, proper consideration of all theoretical considerations will yield an acceptable analysis. But consistency with the data seems to be a useful safeguard against omitting relevant details.

This I do not see:

Also, he is more generous concerning the validity of the assumptions than I am.

I think we share pretty much an identical point-of-view here.

However, here we are of little different opinions (in detail, not in general!):

In such cases the approximate solution to the right problem (provided by non-parametric methods) is often more useful than the exact solution to the wrong problem (provided by parametric models).

Here I disagree with your referrals to non-parametric and parametric models. It is not the case that non-parametric models automatically addressing the right problem. Not at all, to my opinion. I already have a problem with the meaning of non-parametric models. What is modeled there? Typically, these are "non-models", trying to circumvent a proper modelling. What I often see is that people ask for a difference in means (given they have their qualified reasons!), they see that the normal assumption may not be reasonable in light of the given data, and so they perform some Wilcoxon test as a kneejerk action. But then they abandon any insight into the mean difference, and they test something they actually did not intend (i.e. the distributions instead of the means). Bootstrapping the mean difference would be a non-parametric option I'd consider the correct solution still adressing the intended question.

But maybe I am missing something. You may certainly clarify it.

Now you can see from all these answers that between your two groups strain A and B you can use t-test because normal distribution is just an assumption and you can use

Mann-Whitney. Then write t (df) = xxx, p

Dear Béatrice, I am afraid that you still did not get my point:there is no strain difference

Assumptions are not umimportant, just because they are assumptions. They are important. They must be reasonable, sensible, and they must not be obviousely contratidet by the available data.

The assumptions for a t-test are three:

The first two assumptions lead to the often proposed assumption that the values (or, better: residuals) should be normally distributed. The third assumption can be rephrased as: "the applied model must be correct and contain all influential variables", in case of the t-test maybe as: "the used data must be adjusted for all known influential variables".

And this is the crucial point here. Stefan has good reasons to believe that assumption #3 is not sensible, and he further knows the influential variable. It would be rather stupid not to use this information.

Further, the Mann-Withney test is not really an alternative to the t-test. There is no alternative. The MWT compared distributions, the TT compared means. That is a quite different story. Finally, "there is no strain difference" is not a statistical null hypothesis. You must specify what exactly is hypothesized to be identical. This must be something you can measure,

But even when ignoring this, Mann-Whitney is still not ideal and the influence of "day" still covers the important information (also MWT assumes independent data!). To give a numerical example as a proof-of-principle:

Day 1:

Group A:  5.10 5.51 4.56 5.44 5.75

Group B: 5.52 5.32 7.04 7.02 6.74

Day2:

Group A: 10.66 9.25 11.07 7.55 9.75

Group B: 10.71 12.04 11.33 12.05 10.15

t-test: p=0.27

MW-test: p=0.19

"adjusted" t-test*: p=0.005

*linear model with day as additiona explanatory variable

"there is no strain difference" Aha you got his data and could do the computaion, then of course there was an idea to go one step further but this step though blurs the basic idea with the use of selected mouse strains..

It is rarely the case when you have gone through the process of selecting two mouse strains that you would test for no strain difference but a difference between days on a task. The main point is HOW and WHY and WHEN and on which BASIS along with which associated features the strain differ or HOW compared to the normal strain from which the two strains are derived and selected.. Statistics is just a tool in biological psychology.

The Linear model may be just a decoration in the paper, helas!

Jochen,

[...] this is exactly what I tried to say:

[...] to argue that the assumptions should be reasonable, driven by theoretical considerations, and consistent with the data at hand. In principle, proper consideration of all theoretical considerations will yield an acceptable analysis. But consistency with the data seems to be a useful safeguard against omitting relevant details.

Any reasonable person could understand what you meant to say. But, I'm not necessarily a reasonable person and it wasn't exactly what you said. :)

This I do not see:

Also, he is more generous concerning the validity of the assumptions than I am.

I think we share pretty much an identical point-of-view here.

My implicit claim here is that in practice one very rarely encounters the case where the assumptions leading to normally distributed errors are reasonable, consistent with a strong theoretical analysis, and consistent with the data. It's a more pessimistic version of the old adage, "All models are wrong, but some models are useful." As a neuroscientist, I might be too close to the (relatively) recent crisis in the lack of reproducibility in biological research to make an unbiased assessment of the quality the statistical analysis in the average paper; my bayesian prior is that the model is wrong.

In such cases the approximate solution to the right problem (provided by non-parametric methods) is often more useful than the exact solution to the wrong problem (provided by parametric models).

I was trying to be too clever here. It's a rough paraphrasing of a quote by Tukey. In the proper context, it seems to me that he was more or less correct. It was my fault for not providing sufficient context. You are absolutely correct that simply being a non-parametric test is not enough to make a method correct. One still needs to choose an appropriate non-parametric method. And even in that case, there is a convenience fee: when the assumptions are correct the parametric models almost surely provide more accurate estimates. In other words, if the assumptions are reliable a non-parametric test discards useful information. However, if the assumptions are not reliable, then the increase in precision is merely an illusion.

Incidentally, your example is an illustration of why I often recommend bootstrap resampling: it (usually) addresses the intended question and is almost idiot-proof. Bootstrap resampling would be almost magical if not for the associated "convenience fee" of computational complexity and discarding useful information provided by well founded assumptions.