Dear Alexander,

using the NHST approach in replication studies is not the best option. I would suggest using a meta-analytic approach: focus on effect size and confidence intervals rather than p-values. You could also think to use a Bayesian approach

I agree with Mr. Gini – look at the original effect size and use an adequately powered sample for replication. Also, I would not use one-tailed tests post hoc...

I suggest using both 1) one-tailed test together with 2) a meta-anlytic approach

No, you don't have to use a one-tailed test. And eben when using a one-tailed test, the next question would be which significance level are you applying. Due to the symmetry in your hypothesis test, there is no difference between using a two-tailed test and alpha as significance level and using a one-tailed test with alpha/2 as significance level. Knowing the result of the first study turns the problem inherently one-tailed. But having seen the significance in the first study also points you towards using a stricter significance level. Hence, if you really want to take the prior information of the first study into account, you should use a Bayesian approach. Alternatively, if your main focus is on reproducing the original result, you can play the ignorant. If you are really interested in finding out whether the two studies tell the same story, you should compare all study design details: sample size, effect size and confidence intervals. A full meta-analysis approach (Bayesian or not) is technically feasible, but usually you apply meta-analysis for a number of studies and nit just two, so I wouldn't go for that.

Alexander -

Hi. - The advice Gianluca gave you to use confidence intervals is also my preference, whenever possible. There is a great deal of misinterpretation that generally occurs using p-values. (See attached.) Confidence intervals may sometimes be technically misinterpreted, but for practical purposes of decision making, they are much better than hypothesis or significance tests.

In general, by the way, in cases where you do not think you can use a normal or t distribution for a confidence interval, other distributions can be applied. In any event, you could apply Chebychev's inequality. Even just noting a point estimate and its standard error can lead to a better decision than the use of a single p-value.

I did, at one time, do a lot of work on the Fisher Exact Test power analysis, but without such an evaluation of a second hypothesis, you do not have a great deal of information, and at any rate, it is far harder to interpret/communicate clearly to others. If you can express what you need to express with confidence intervals, that would be much better.

Jim

Article Practical Interpretation of Hypothesis Tests - letter to the...

Thanks so much for your interesting comments! To further illustrate the problem, here are some data (there are more, also poiting into the same direction):

Study 1 (published):

Control: 33 sick / 27 healthy (33/27)

Treated: 45/13

Fisher's exact test result: p < 0.05 (two-tailed)

Own study:

Control: 81/15

Treated: 92/4

Fisher's exact test result: p < 0.05 (two-tailed), p < 0.01 (one-tailed)

Odds ratios:

Study one: 2.83 (1.273 to 6.303) 95% CI

Own study: 4.26, (1.358 to 13.36) 95%CI

Combined: 2.97 (1.604 to 5.497) 95% CI

Best,

Alexander

Thanks so much Paul. It seems indeed that I have to cross the border to ODA land ...

I can't open the file, could you convert it into pdf or word or excel ...?

Thanks again! Alexander

Paul,

Thanks so much for your help and the analysis, it is greatly appreciated! Now I know a good deal more about the way how I can analysze may data.

I meant "my" data, not "may" data ... (sounds like mayday...)

Is there a distributor for the software in Germany?