The underlying misconception is: "significant p values" do not tell you a story. At best, they may help to render a previousely elaborated story (or a part of it) unreasonable or incomplete.

P values are "shadows". They may tell you if your idea from the object that throws this shadow is unreasonable. But is is not sensible to induce from the shape of the shadow to the shape of the object. Turning the object around will give a different shadow - this is what you encountered - and this can make all your nice inferences about the object meaningless (well, not make them meaningless - is only shows that they are essentially meaningless).

So first you have a story (this includes having an idea about the whole structure of the statistical model). Then you can use p-values to see if some parts of the model are not well supported by the data you have.

p-values and significance is NOT A TOOL to create a story.

ANOVA (Analysis of variance) test is basically applied for hypothesis testing involving samples taken from more than two populations having different or same population means. There are two types of ANOVA (one way and two way, 'n way ANOVA is also there). In one way ANOVA we study the hypothesis from the perspective of only one factors whereas in two way ANOVA we take into considerations two different factors of the Populations. So different group effects have to be there.

Thanks Sovik - I understand that but still don't see why my group effects should be different if the same groups are used - the only difference in the 2-way is that as well as a group designation (test/sham/naive), there is also a hi/lo phenotype descriptor, essentially 6 groups.

My idea of doing the 2-way was to see whether the difference I saw in 1-way is influenced by this second factor. When I look at the stats output, it shows me first the hi-lo effect, then the group effect, then the interaction. 

Am I wrong in thinking the group effect would be similar to 1-way i.e. the 2-way is 2 1-ways with an additional interaction anova?

It sounds like there is confounding in your data.

For example, a data set I used "showed" that people that drink heavily have a higher rate of cancer. When you add in the factor "Smoking" you see that there is high correlation between smoking status and heavy drinking. The true effect is between smoking and cancer. 

Something you can try,  create 6 groups 1 to 6. Group A lo, Group A hi, Group B low, Group B hi, ... And see what those results are with a 1-way ANOVA. If his method and the 2-way ANOVA give you the same results you know what happened. 

The underlying misconception is: "significant p values" do not tell you a story. At best, they may help to render a previousely elaborated story (or a part of it) unreasonable or incomplete.

P values are "shadows". They may tell you if your idea from the object that throws this shadow is unreasonable. But is is not sensible to induce from the shape of the shadow to the shape of the object. Turning the object around will give a different shadow - this is what you encountered - and this can make all your nice inferences about the object meaningless (well, not make them meaningless - is only shows that they are essentially meaningless).

So first you have a story (this includes having an idea about the whole structure of the statistical model). Then you can use p-values to see if some parts of the model are not well supported by the data you have.

p-values and significance is NOT A TOOL to create a story.

Thanks Andrew & Jochen - I was getting a bit tied up in it all as there was also a mis-labelled bit of data on one set, which I found and fixed and now it seems to make more sense.

I've had a few comments in response to this query elsewhere which have suggested using a factor based on an outcome measure is not good practice - do you have any thoughts on this? I'm currently writing up this work for a publication and also for my thesis, so was going to leave out the responder analysis from the publication, but include it in my thesis.

My 'story' has always been that the intervention would cause a certain change. My theory using the hi/lo was to investigate whether

a) accounting for phenotype explains variation observed

b) the phenotype observed in one outcome is seen in the other 3 related outcomes (single assay - 4 outcome measures, primary responders used to examine relationship between outcomes)

Jochen - you say that p-values are not the whole story. This is something I have long suspected, and why I now present data as dotplots rather than solid bar charts when possible.

However, how do I explain what I am seeing when commenting on 'non-significant' trends is frowned upon? Thoughts very much appreciated!

I don't have an ultimate solution to this problem. Many researchers/bosses/reviewers/editors are simply very indoctrinated by the dichotomous system of accepting and rejecting hypotheses, and referencing a list of hundreds of papers pointing to the difficulties and the unreasonabilities of this approach in research won't help.

The way is not easy and it forces at first you to have a sound understanding on the principles and the philosophies of estimation and of hypothesis testing. This is the most difficult and the most important part.

In short: there is nothing like a free luch...

Based on this knowledge you can then demonstrate that your work is explorative (not confirmatory), and focuses on estimation of well-defined effects. There is no p-value that needs to be calculated at all, and there is no distinction between (statistically) significant and non-significant results. Results are interpreted based on the effect size estimates and the obtained precisions of these estimates. In your words it would mean that you can not just say "there is a trend" - you must say how strong the trend is and you must judge, scientifically, based on expert understanding, if a trend of this strength is relevant to the problem. This is not easy.

Finally, when others insist on having a p-value and a significance rating for your results, then I would ask what precisely the p-value or a significant result exactly means. There are two possible types of answers that you can get then: either the given meanings are wrong (what you would have to explain why they are wrong), or they are in fact correct but then the answers will make obvious that p-values or "significance" is nothing that helps you in sensibly interpreting your results.

The worst one can do is to follow the Neymanian paradigm of hypothesis testing (with a fixed alpha), at the same time not defining relevant alternatives (a procedure that is often termed "null hypothesis significance testing") and then to interpret "non-significant" results as "trends" or something. This is nothing but a big mess (there are many reasons, e.g. Neyman is not for inference about particular results - it is about decisions, not about science; the value of alpha is not justified by any reasonable agrumentation; without specifying a relevant alternative and an appropriate power even this decision-philosophy completely breaks down logically). Unfortunately, I read this very very often in papers and theses.

So a first step for you will be to read and learn what Neyman really wrote about hypothesis tests, and what Fisher really wrote about significance tests, and then to understand what these procedures really can do (and what they can not do) why these two procedures are logically incompatible (you will then notice that the text-book approach of "null-hypothesis-significance-testing" most students learn is logically severely flawed). You will find to learn that most statistics is about data, given some hypothesis underlying the data-generating process. You will see that this is very unsatisfactory, because the data is the part you already know (you measured it!) and the hypothesis is the thing that is unclear or uncertain. This problem is known since long time and Bayes was the first to present a solution to this problem. However, this was mathematically intractable for a long time and also raised concerns about objectivity. You will find that the "large sample theory" coming from Gauss and others was not only to get likelihood functions that approximate the normal distribution but that the likelihood function becomes a good approximation of the posterior, too (up to a constant proportionality factor). So there is a lot to learn and understand about epistemiology, decision-theory, quality control, the frequentistic and the Bayesian philosophy of statistics and much more.

As I said: there is no free lunch...

If I could recommend twice I would - thanks so much for that!

I have been involved in discussions with my boss over confirmatory and exploratory research, and he feels that the difference between the two is indeed the use of statistics (descriptive vs inferential) and interpretation, rather than the methods used (as has also been suggested). Lots for me to think about!

A big problem is that "inferential statistics" is often used sysnonimous with "hypothesis testing". This is dangerous because it can be quite misleading.

Inference is about learning something, about getting beyond things we know, specifically getting beyond the observed data. This process of induction can therefor not be solely based on data. There must be an idea, or a model, coming from somewhere else. This is a creative process that can not be dealt with any mathematical/statistical tools. However, and this is important!, statistical tools can then help us to see how and how well the available (known) data "fits" to the model, how well it can be "explained" by the model, at tell us what we can reasonably expect, given we knew the values of the predictors.

Popper made clear that the correctness of a model can not be assessed logically by the accordance to data. A model may still be wrong even if all available data are perfectly fitting. From the logical point of view, a model can thus not be verified by empirical data. But it can be falsified, since a single non-fitting observation would proof that the model is wrong. Thus, we can only "learn" that a model is wrong. This had a large influence on science, and people thought that the attempt to "falsify" a model is the only way to learn something.

This was adopted to statistical models by defining a specific hypothesis (related to the model) that might be falsified by the data (-> rejection of the null hypothesis). The problem here is that the data are not logical statements that are either true or false, as required for the Popperian logic. There is uncertainty involved, and for any given hypothesis the data can be just more or less likely (but never be "wrong" or "impossible"). So arbitrariness enters the field when we have to define how "unlikely" is "unlikely enough" to consider a hypothesis falsified.

But falsifying (rejecting) hypothesis does still not tell us anything new. So this is not an inferential process, and a hypothesis test is actually not an inferential procedure. The inference happens when the model is set up or invented based on some idea of an expert. The test will then only "sort out" models for which the available data is relatively unlikely.

Here the philosophies of Fisher and Neyman separate. Fisher used the significance (a continuous! measure relation data to a model) to judge if is is feasable to continue the work on a model, what neccesarily includes a subjective aspect (integrating the available knowledge and, possibly, personal beliefs and hopes and aims). Neyman tries to circumvent the subjective aspects and developed an entirely decision-theoretic approach of hypothesis testing. This allowed to "objectively" control error rates (that still have to be chosen based on largely subjective reasoning!) and this at the cost of sensibly judging the result of a single experiment (what, to my opinion, is one of the most important steps of the inferential process in research). For Fisher, significance testing is not related to inference; it is only a last resource for a researcher to find out if a further work on the model might be promising (or rather hopeless). In contrast, Neyman built a soul-free" system of decision-making under uncertainty that allows a "statistical inference" in the sense that not any particular result is important but that on average and in the long run (i.e. "statistically"!) the research community will accumulate correct models and thin out wrong models, based on automatic accept-reject rules under control of error-rates. However, reasonable or sensible error-rates can hardly be defined in a research setting (that is different for the control of, say, industrial processes, where the expected wins and losses can be quantified in monetary units). Claiming that "p