I've performed analyses on a series of variables, 3 of which yielded 100% sensitivity, 100% specificity, 100% positive predictive value, 100% negative predictive value, AUC of 1.0 (1.0,1.0). I did a histogram analysis of the group with the disease outcome and that without - there is clear separation with no overlap. Because I was skeptical, I ran all kinds of sensitivity analyses (subgroup - by risk categories, by ethnicity - still ended up with similar results although some slightly ablated). All models were adjusted for potential confounders. Seemed too good to be true. My sample size is over 2000 - could it be significant by virtue of high power? My Hosmer-Lemeshow Goodness of Fit test was not significant.
How often do we see perfect or near-perfect diagnostic measures? I checked different sources and all seem to point to the 'textbook ideal world' case but don't shed light on what range of values is typically seen.
Most textbook discussions, it seems to me, do say that perfect discrimination is only rarely (if ever) achieved in the real world. One reason for that is that diagnostic tests are typically used when there is diagnostic uncertainty. And in that situation, the distributions for those with and without the disease/condition have considerable overlap--so false positives and false negatives will both occur with some frequency.
In your case, I wonder how the subjects were selected. Are they a random sample from some population where there is diagnostic uncertainty? Or did you select a sample of folks who clearly have the disease and another sample of healthy controls who clearly do not have it? If you did the latter, then your study suffers from spectrum bias (see link below), and overestimates the sensitivity & specificity if the test is to be used when there is diagnostic uncertainty.
HTH.
http://www.ncbi.nlm.nih.gov/pmc/articles/PMC1188226/
Most textbook discussions, it seems to me, do say that perfect discrimination is only rarely (if ever) achieved in the real world. One reason for that is that diagnostic tests are typically used when there is diagnostic uncertainty. And in that situation, the distributions for those with and without the disease/condition have considerable overlap--so false positives and false negatives will both occur with some frequency.
In your case, I wonder how the subjects were selected. Are they a random sample from some population where there is diagnostic uncertainty? Or did you select a sample of folks who clearly have the disease and another sample of healthy controls who clearly do not have it? If you did the latter, then your study suffers from spectrum bias (see link below), and overestimates the sensitivity & specificity if the test is to be used when there is diagnostic uncertainty.
HTH.
http://www.ncbi.nlm.nih.gov/pmc/articles/PMC1188226/
Thanks Bruce. My diseased cohort were selected because they had the disease - it is a very rare condition so we collected patients from many clinics. The rest I chose a spectrum of participants who were from a population-based cohort study. I selected a range who were low, moderate and high risk for a similar type of disease that is more common to that in question. Unfortunately, because we know so little about this rare condition, it is difficult to find people who may be on their way to developing it - thus the whole point of developing this diagnostic marker.
Error is inherent to the process of dianosis. It is unlikely that you will have a perfect dianostic measure. You wrote about people who may be on their way to developing the outcome, so I assume it might be chronic. From a theoretical point, you can ask when someone begin "having" the disease you are interested in. The answer might not be clear, and this is one of the reasons why a perfect diagnostic test is unlikely.
One possible point of discussion is the difference between the real disease and your measurement of disease. Even gold-standards tests are not perfect - that's why eventually they get replaced or improved. The way I see it, you have a test that measures closely what you assume to be the gold-standard on the diagnosis of your disease. However, this is extremelly valuable, specially if your test is cheaper/quicker/simpler than the gold-standard.
Another quick point is relative to your cases. Since you recruited them from clinics, they might have a more severe clinical manifestation, which might make easiear to separate them from the non-diseased people. Try running your test on a harder to diagnose population and check how it performs.
Imperfection is highly improbable. You have to study a wider audience (normals, possible confounding diagnoses, etc.). I hope you have also challenged the environment of the testing (temperature, humidity, sample storage and age, etc.).
Because you are investigating a rare condition it is even closer to very improbable to get a near-perfect test. Perfection (i.e 100% in everything) is the impossible dream, because there is and it always be a biological variability that nothing can encompass. It is weird having 100% sensitivity AND specificity. All assay designers and producers are trading off one of them against the other, depending on the intended use of the test. If the screening / detection of a certain parameter is the goal, you may prioritize the sensitivity and for sure you will trade off the specificity.
On the other hand, PPV and NPV are not intrinsic to the test; they depend on the prevalence of the (medical) condition, actually refer only to a predisposition/risk of having or developing that condition. For certain tests, PPV is more relevant than NPV. PPV in a contingent population appears improved because you have already a selection.
Your cohort should be general population, and size should be at least the minimum prevalence of the disease your are looking for, i.e 1:10,000 = 10,000, with complete follow-up for outcome. For the moment you know for sure that the biomarker works for ill people. Do you know the same for the rest of your population? Assuming they were negative, did you finish a follow up to see if that negativity is real?
What is most important for this test? to diagnose patients in general? at risk patients in particular? to screen for the risk of developing/having the condition? After defining the goal you will be able to figure out the most relevant parameters to calculate for the test and see that perfection % fade inversely proportional with increasing the number of unfiltered population investigated.
- Badges
- Science method