In the attached study which seeks to identify immunological markers associated with Chronic Fatigue Syndrome, the authors use sera from 21 cases and 21 controls, and screen it with a 125,000 random 12-mer array. They identify a 25 peptide signature that is specific (93%) and sensitive (98%) for CFS. These peptides relate to self-antigens and endogenous retroviruses, and a small but eclectic mix of bacterial and viral proteins.
Could a statistician explain how you can demonstrate that (in this and any other comics studies looking for biomarkers or signatures associated with a specific condition or outcome), of all the trillions of possible signatures, this observation of a 25 peptide signature is not just a random observation. If you anonymised cases and controls and randomly assigned them into one of two groups, and maybe bootstrapped this 100 times, I think that there are so many possible signatures (combinations of 1,2,3,4,5...25...n peptides, among a total population of 125,000), that with such a small sample of just 42 participants, you will likely always identify some kind of differential signature between two random groups.
In this kind of analysis how is significance testing done, and how does it account for the very high total number of possible observations that would be deemed significant?
To all readers -
I believe this response is correct, but if anyone knows it is deficient in some way please notify me, as I would like to continue to learn about questions like these.
Hi Matthew,
Thanks for the thought provoking question. Interestingly, your framing of the question suggests one possible answer. Permutation based testing, which is very similar to what you are describing anyway, can be used to generate an empirical p-value to quantify the likelihood of observing events (in this case combinations of biomarkers) like these.
Suppose you conduct 1,000,000 permutations (shuffles) of the case and control labels (with 12 and 12 this might not be possible, but for arguments sake suppose you can) in the manner you suggest. You can then measure the total number of times the random shuffling procedure produced a set of N biomarkers with equal or better discriminatory power than the combination of biomarkers initally found with the case and control statuses correctly assigned. (it is worth noting that you would need to use some clever heuristics to write a program fast enough to do this 1M times, as searching through in a rote fashion would not be feasible, and more importantly that this heuristic would need to be the same heuristic that you used to find the initial combination of biomarkers that is being reported).
Suppose that, of the 1,000,000 shuffles, in 2500 random shuffles a combination of biomarkers *at least as* sensitive and specific as the combination of biomarkers that was discovered is found. One could then ascribe an empirical p-value of 2500+1 / 1,000,001 to this.
One could focus on just sensitivity or just specificity and derive a similar result.
Answers based purely on statistical theory are possible here, but a formal treatment would be time consuming. If you seek this, I might try stats.stackexchange.
On a practical note, I agree with you that with 42 participants and 125000 choose 12 combinations of biomarkers, overfitting is the norm rather than the exception. With a larger study, one might be well advised to try cross validation of some kind.
Many thanks Vincent and Nina for your in depth and considered answers. More stats reading for me to do clearly. Any pointers towards good reviews on stats for omics analysis are much appreciated.
Another consideration in these kinds of studies that I rarely see addressed within methods section is possible temporal changes in gene expression or immunological profile in one individual, that may change from day to day.. or even hourly.. based on a range of factors: Was the plasma sample collected at the same time of day for all participants? What had they recently eaten? Had they had any recent infection or illness? Has anyone ever done an expression array on a single healthy subject at multiple time points, and characterised how constant signals are over time? Many thanks all. Matthew
It is the principle of random sampling to get results that won't systematically depend on any factor we can't control or we don't even know. Yet, any concrete sample may well show systematic dependencies, but statistics is about the procedure, not about the observed case. In any case should the sampling strategy be selected based on the aim of the study. If you think that, for instance, males and females will have considerably different possible signatures, you can either decide to sample only females (or males), and your conclusions will be statistically supported only for the generalization to the sex you studies (the generalization to the other sex can be very right or very wrong - due to the chosen sampling rules we have no data to say something about that); or you sample both females and males, and the combined results will be generalizeable to both sexes (at the possible cost of a limited precision) or the subgroups can be analyzed individually, so you may come up with a (more or less) different generalization for females and for males.
- Badges
- Science topic
- Similar topics
- Biological Science
- Microbiology
- Microorganisms
- Bacteria