It is a systemic problem in all research that depends upon p-values. There is no basis to use p-values as a criterion for a conclusion. There are no rules as to how to apply them. As Jochen asked, what is the basis for deciding these are false positives? A p-value cannot be classified as false or true unless you have other information.

A study must start with information. The information leads to a proposition that can be tested. The proposition must be tested in a logical context. The context must include the degree of evidence necessary to make a conclusion. A conclusion can be drawn when the evidence is sufficient. If you need p-values to prove your point then you must qualify the context and evidence that is supported by that p-value. I think you will find that a p-value does not suffice.

How do you know that these are false-positives? If this is clear a priori then why where they tested at all?

Further, pathway analyses must be interpreted with great care. These analyses usually are very crude tools. They do not give more than just a vague hint towrads the direction one may look at. Results may generate hypotheses, not test them. Most analyses also do not account for the relative importance of the genes in a pathway. Some pathways are collection of genes that would are never interacting in the same cell type under the same condition in the same developmental state and so on. Many pathways are abstractions of what is really going on in the cell. It should not be considered a 1:1 relation. Often the data is taken from one time point (ignoring the dynimics), and from cell pools (ignoring different cell states) and from tissue or organ homogenates (ignoring cell-type specificity).

The interpretation of a pathway analyses requires a lot of expertise going far beyond the data. Simply looking at p-values is by far not sufficient for interpretation (and it might be rather misleading!). The "problems" might be solved in future by a better dissection of the specifities of cell-types and developmental stages, also by consideartion of the relative importance of the genes, the crosstalk between different pathways and by a more dynamic modeling (via petri nets towards a real quantitative modeling).

Hi Martine,

Thanks for bringing this up. It is quite a complex topic and the reasons for the “unexpected results ” can have multiple reasons starting with the study design.

a) Case population was selected from participants with epilepsy and control population suffered from pancreatic cancer (unlikely because pancreatic cancer is rare)

b) There is no signature for epilepsy and all you get is noise

c) Results are not properly cross validated (did the authors perform a permutation test? If not that could be a good explanation)

d) Creating pathways and choosing its content is a little bit of a subjective choice and it is not necessarily  taking into account cell type, whole organ, or body expression levels.

How to get around it:

 proper study design

 good data exploration before you do pathway analysis

 include a vigorous cross validation step into your pathway analysis. E.g. use your study data, but randomize your case and control status. If the software still comes up with a significant result than the software package doesn’t work properly, because you provide it with non-sense information and it comes up with significant results. Does that make sense to you?

Is it a systematic problem?

 I think so, but I could be mistaken. it requires a bit of skill to get this right and plug and play software can produce results have to carefully validated.

There could be many other explanations but without any detailed knowledge about the study it is difficult to come up with the correct answer.

Kind regards,

Jens

It is a systemic problem in all research that depends upon p-values. There is no basis to use p-values as a criterion for a conclusion. There are no rules as to how to apply them. As Jochen asked, what is the basis for deciding these are false positives? A p-value cannot be classified as false or true unless you have other information.

A study must start with information. The information leads to a proposition that can be tested. The proposition must be tested in a logical context. The context must include the degree of evidence necessary to make a conclusion. A conclusion can be drawn when the evidence is sufficient. If you need p-values to prove your point then you must qualify the context and evidence that is supported by that p-value. I think you will find that a p-value does not suffice.

Many thanks for your answers. Just to clarify, this was from a previously published study by another group.

Jochen, you raise several important points. Pathways are abstractions, and further knowledge is needed at the cell-specific level, as well as an understanding of what is going on at a particular developmental stage.

For example, the neurotransmitter GABA is inhibitory after birth, but actually excitatory before birth and around the neonatal period, due to a developmental switch in the Cl transporters. So the GABA pathways need to be drawn differently, depending on developmental period being examined.

Your point about the relative importance of a gene is underscored by the observation that a gene which may be "hit" a number of times in a particular disease, while another gene is hit only once, needs to be taken into account. 

Jens, you also raise the important point that creating pathways is a very complex matter. The protein-protein interactions which form the backbone of pathway construction are by no means universally agreed upon. For an example, Mullin et al in a thought-provoking article

Transl Psychiatry (2013)3, e329; doi:10.1038/tp.2013.108

examine 4 different interpretations of protein-protein interactions in a particular set of pathways, all derived using different approaches to data; there are some areas of agreement, but other parts of the pathway have limited consensus. Which is the "right" pathway? Methodology is crucial here, the more-cross validation (by several different methods) there is, the better. If results are buttressed by converging lines of evidence, as they are in the Mullin article, that is a major advance.

Joseph, I agree completely that p-values in isolation are not sufficient.

One other observation I would add is that in the central nervous system, pathway analysis will be very complex for a set of additional reasons. One is that CNS proteins are often extremely large, and have binding sites not only for proteins, but small molecules and lipids as well. Some have various binding sites for all 3 types of interactants  eg the various GABA and glutamate receptors. So protein-protein interactions capture some, but not all of the biological interactions of these molecules.

Clinical grade pathway analysis in the CNS is likely to be extremely challenging! 

A lot of really good and interesting points being made which will help me with future pathway analysis.

One additional thing that I would add for future readers, would be to look at which genes/proteins are being found within a pathway. E.g. do annotations 1 and 11 in the original post actually contain the same epilepsy related-genes?

In a hypothetical situation, it could be that a subgroup of epilepsy genes are involved in control of the cell cycle, and this subgroup is resulting in the cancer related annotations seen. Therefore these annotations could be telling you something interesting, but need further interpretation to get the biological meaning. The reason a broader cell cycle annotation isn't enriched could be because they are only a specific subtype of cell cycle genes which have not been given their own annotation yet.

To reiterate what other have already said, it’s important to not just trust whatever answer the tools give but to use your knowledge to interpret them.

Martine -

I am in agreement with cautions mentioned above, regarding p-values. In addition, they are functions of sample size, which can generate some of their problems in interpretation. That is discussed in the short note and the RG 'abstract' at the link attached. But as mentioned above, you have quite a few other things to consider.

Jim

Article Practical Interpretation of Hypothesis Tests - letter to the...

Thank you, Jim for your comment. I will read your attachment carefully and get back to you.

Meanwhile, an excellent recent article discusses the issue of cellular heterogeneity in the CNS, with its attendant challenges and potential for the integration of the 'omics. This speaks to your point, Jochem about different cell types/markers in the CNS; as you point out, there are developmental shifts as well. The article outlines a roadmap for neuroproteomics- and also why it is important to integrate knowledge from the different 'omics, since a complex system like the mammalian brain needs to be looked at from different levels (hierarchies) of analysis. Each level of analysis has its strengths and weaknesses; they complement each other.Decoding neuroproteomics: integrating the genome, translatome and functional anataomy etc Kitchen et al  Nature Neuroscience online 28 Oct 2014 doi:10.1038/nn.3829

Jim - I very much appreciate your input, but I am not a statistician. This is why a multidisciplinary forum like ResearchGate is so valuable. Could you clarify for me what you are stating, and what predictions you would be making, so I can get the gist of what you are proposing? 

Perhaps I could add that in clinical research, it is recognized that the p values become may become more significant with an increased sample size of well defined (relatively homogeneous populations of patients). An example would be GWAS (Gene Wide Association Studies) of neuropsychiatric disease, where significant p values were attained only by studying thousands of patients. The p values may also become less significant  if heterogeneous collections of patients are being studied. However, it is not always apparent ahead of time how to "slice and dice" subpopulations of patients. 

I notice in an otherwise excellent  paper by a reputable group, the following results in regard to Bipolar Disorder, a neuropsychiatric condition, that their pathway analysis yielded the following in Table 1: (their top 10 pathways, listed by p value) 

      1)   1.01 × 10−6    0.005    GO:51568        Histone H3-K4 methylation                        

      2)   3.82 × 10−5   0.093    path:hsa05218     Melanoma                                         

       3)   1.16 × 10−4    0.093    GO:7129 Chromosomal synapsis genes                                     

       4)  1.27 × 10-4      0.093    path:hsa05213     Endometrial cancer

       5)  1.34 × 10−4    0.093  P00003 Alzheimer disease–amyloid secretase pathway      

       6) 1.35 × 10−4    0.093    path:hsa05215     Prostate cancer                                  

       7)  1.50 × 10−4    0.093    path:hsa05216     Thyroid cancer                                  

        8)    1.59 × 10−4    0.093    GO:90066          Regulation of anatomical structure size           

         9)   1.81 × 10−4    0.093    path:hsa05214     Glioma

         10)    1.87 × 10−4    0.093    GO:70192         Chromosome organization involved in meiosis    

I can see the potential relationship of glioma, a brain tumor (ranked 9th), but I fail to see the relationship between  bipolar disease and the more highly ranked melanoma (2nd), endometrial cancer (4), prostate cancer(6)  and thyroid cancer (7).  I can see a potential relationship with some of the other ranked pathways. But why do several tumors not even in the CNS rank so high? Something is amiss here.

Nature Neuroscience 2015 doi:10.1038/nn.3922