Say that I have two sets of data, each with 3 biological repeats (unpaired).
Dataset1:
control:0.27/0.376/0.393
treatmentA: 6.285/7.574/18.171
unpaired two tailed t-test P = 0.051 > alpha 0.05, not significant
Dataset2:
control: 0.262/0.217/0.355
treatmentB: 0.401/0.449/0.393
unpaired two tailed t-test P = 0.036 < 0.05, significant
If you ask me, treatmentA is obviously better than B, but with a very slight chance it can be worse than control. treatmentB is equivalent to "drink-more-water-and-you-will-feel-better", but it is significant.
Somehow, in terms of publishing, everyone boos at treatment A and cheers at treatment B, simply because of the "*" ?
So if my data end up like treatmentA (dataset1) , other than just take the D and report: "there is no significant difference between treatment A and control", is there anything I can do to make it sound more exciting (OF CAUSE NEED TO BE WITHIN THE SCIENTIFIC ETHICAL BOUNDARIES)?
Is it possible to use one tailed t-test instead and say we only focus on increase etc.
I am not savvy with statistics, but any other model may be a better solution?
First of all, never write things like "there is no significant difference between treatment A and control". This is a common but very bad style prone to severe misunderstanding. It is better to write "the (sample) difference between A and control was not statistically significant different from zero", what means that the observed sample difference is statistically compatible (or not yet "too incompatible") with a model assuming a population difference of zero.
Second, I am quite sure that the response you are analyzing is a strictly positive quantity (like a concentration, an intensity, an expression value, something like this). Assuming that such variables are normal distributed with a constant variance being independent of the mean is ... stupid ..., well, at least it's very dangerous. If my feeling is correct, then you should either do the test using the (log-)transformed response or use a test (or model) based on more appropriate assumptions (e.g. a GLM of the gamma- or quasi-Poisson family of distributions).
Third, if a comparison of the effects of (A vs controlA) against (B vs controlB) is of interest, then use a model in which you can actually test this "difference of difference". Do not speculate based on the different p-values you find in the isolated analyses. The two p-values don't give you any information about the statistical significance of their difference (or of the "difference in difference" of the response). Use a model including all the data, including an appropriate interaction term, and accounting for possible systematic differences between the two data sets / experiments.
(Fourth: a sample size of n = 3 per group is really really tiny. One should not expect to get anything more than a crude first impression, irrespective of the p-values, which can be grossly wrong in such tiny samples. This is okay as one piece of evidence in a larger, coherent story with more data from more experiments all around the same biological process. If this was the only data to make a major point, then the data is extremely weak - again: no matter what the p-value are!)
Applying all (except point 4) above to your data, assuming that the experiments behind the two data sets are comparable(*), gives me following results:
Treatment-Effect of A, on log-scale: 3.3 (0.27), p < 0.001
Treatment-Effect of B, on log-scale: 0.4 (0.27), p < 0.15
Difference between treatment effects (B-A), on log-sclae: -2.9 (0.38), p < 0.001
The values in parentheses are the standard errors of the estimates.
The conclusion would be that the data provide sufficient evidence to claim a positive treatment effect of A, but it does not provide sufficient evidence to conclude a positive treatment effect of B. However, it does provide sufficient evidence that the (positive) effect of A is larger (more positive) than the effect of B (of which we cannot even say if this is positive or negative).
It might be more interesting to provide confidence intervals as proxies for hypothetical effects that are statistically compatible with the data. Based on that information one might get an idea if a treatment would be worth in practice (what would still require some additional knowledge or assumptions about the loss-function; what does the treatment "cost" (including possible side-effects), how much "loss" can be saved by applying the treatment, is there a remaining expected "net-loss"? Should one consider a more pessimistic or a more optimistic estimate (-> referring to the lower and upper limits of the confidence interval)?).
(*) A critical point still is if the effect itself can be confounded with the experiment. If this is possible, then it is impossible to say which treatment is more effective. There is no statistical way to decide that.
PS: Simply throwing a t-test on data is NOT doing statistics! ;)
"Is it possible to use one tailed t-test instead and say we only focus on increase etc." - This could have been permissible if you made this decision before you made the experiment, conduct the stats and saw the results (could lead to bias). By using one-tailed t-test, you are lowering the evidentiary standard/accepting a smaller effect size difference when this should not be the case. For scenarios and when it is acceptable to do 1-tailed t tests see https://statisticsbyjim.com/hypothesis-testing/use-one-tailed-tests/.
"If you ask me, treatmentA is obviously better than B, but with a very slight chance it can be worse than control. treatmentB is equivalent to "drink-more-water-and-you-will-feel-better", but it is significant." - Jochen has nice pointers about this but I am wondering why you didn't run an ANOVA if you are wondering whether treatment A is better treatment than treatment B?
- Badges
- Science method