I have 1600 drugs to treat a condition. I firstly test them each by performing a drug screen. I divide the 1600 drugs into successes and failures.
For example:
100 drugs were successful
1500 drugs were failures (had no effect).
I then have an in silico predicting model, to which I apply all 1600 drugs. Again, like the experimental drug screens, the predicting model will yield successful drugs and failure drugs.
For example:
X drugs were successful
1600-X drugs were failures
I want to know what kind of a statistical test I can perform that will return some measure of significance between the two methods used. So I can say whether both reproduce very similar success/failure outcomes (sets of drugs)?
Thanks
If I understand the problem correctly, you have two methods of evaluating the same "subjects" (i.e., drugs). Each method assigns an outcome ("works" or "does not work") to each drug. Thus the results of the two methods appear to be dependent, and so you should use McNemar's test for equality of proportions.
It appears that you are dealing with Binary data.
Binary data can have only two values. If you can place an observation into only two categories, you have a binary variable. For example, Success or failure.
Binary data are useful for calculating proportions or percentages, such as the proportion of success in a sample. You simply take the number of success divided by the sample size. Hypothesis tests that assess proportions require binary data and allow you to use sample data to make inferences about the proportions of populations.
Proportions test to compare two samples
Now you will make a decision based on the proportions of success. The goal is to determine whether the two methods produce different proportions of success.
To make this determination, we’ll use the 2 Proportions test. For this test, the hypotheses are as follows:
- Null hypothesis: The proportions of success for the two populations are equal.
- Alternative hypothesis: The proportions of successfor the two populations are different.
- Likewise u can proceed with failure also.
Dear Ajit,
Thank you for your response. That is exactly what I needed to know. I've found at how to do this and I've applied a few test scenarios to see if it works with my data. I used a two-tail test statistics to see if the proportions of "failed drugs" via experimental and via in silico, were statistically similar (H0) or not (Ha). Works perfectly.
The only issue I now have is one slightly more complicated. That is the power of each technique over the population sizes. How about whether the proportion of drugs successful/failure are to be matched?
For example, at the moment I am just testing the proportion of failed/successful drugs via experimental and insilico. If the proportions are identical then Z (test statistic) is 0. But what if the proportions, although identical, are actually pulling up different drugs e.g.,
10 failed drugs via in silico and via experimenta. However:
in silico drugs: A to J.
experimental drugs: K to T.
Many thanks
If I understand the problem correctly, you have two methods of evaluating the same "subjects" (i.e., drugs). Each method assigns an outcome ("works" or "does not work") to each drug. Thus the results of the two methods appear to be dependent, and so you should use McNemar's test for equality of proportions.
I totally agree with Joseph. Kappa statistics may help also, which shows the degree of agreement beyond chance.
I suggest para-phrasing the title of the question, as it is not clear whether your aim is comparing the two populations or the two methods!
If you are assessing predictive ability of the model, ROC area or concordance probability has been a preferred approach for estimating pure index of predictive discrimination for a single model; however, it is not useful for comparing two models. Percent classified correctly has many issues with improper scoring rule, which will lead to incorrect model. There are different ways to validate the predictive model, and I would suggest that you work closely with a biostat colleague to build and validate the model. 'Applied Predictive Modeling' by Max Kuhn is an excellent reference if you wish to learn more about predictive modeling.
It is not clear if you are testing the populations or the methods. The population is 1600 drugs. The method can be either in silico or screen. Furthermore it would seem that you have only one replicate. To further test the method you will need other populations of drugs, or at least replicate the existing experiment at other locations (different hospitals/cities/countries).
In the screen you have a binary outcome. However, it is not clear if the data are binary. If I have 1600 drugs and 1600 subjects I could expose one subject to one drug. I have now confounded the subject response with the drug performance. I am more interested in the percentage of subjects that respond to the drug rather than any one outcome. I could have several tens of thousands of subjects, or I could expose subjects to multiple drugs (with some compensation for the risk of drug interaction).
Hi anthony. In dealing with determining difference between two populations where the responses gathered are dichotomous (success or fail), I suggest the use of Z-test on Two Proportion or Z-test on one Proportion. In z-test on one proportion, your null will be : equal proportion between success and failure. Alternative hypothesis would depend on your research hypothesis: whether you are assuming that the proportion of one outcome would be greater/lesser than the other (one-tailed) OR the proportion of the two outcomes would not be equal (two-tailed).
The McNemar test is o.k. for testing equal success rate (vs. not so) - as already discussed. An ordinary 2by2 test (e.g., Fisher's Exact Test) would of course examine whether the two success criteria look completely unrelated (in which case Anthony would hardly be working with these data).
Apart from that, I feel that Anthony's problem is of a different kind, which does not ask for a signif. test because no relevant H0 is on the agenda: Do the 100 bio-successful drugs and the X (=, say, 108) silico-successful ones resemble each other well (i.e., are the two subsets fairly/sufficiently/acceptably similar and overlapping) or not? This obviously depends on the definition of 'similar' - and there is no relevant null hypothesis. (The H0: Perfect Agreement is already rejected because 100=/=108). There is only a similarity desideratum, which (in principle) must be made explicit in numerical terms prior to the data analysis.
kappa may be o.k., but (in principle) only if the two characterization processes are on the same footing - rather than one [silico] being seen as a predictor other [the biological TRUTH]. In that latter case, you have diagnostic problem (symptom in silico; underlying disease found in the drug's biological performance).
If I have understood then it can be done by test of proportion using binomial test in case of small sample and using normality test in case of large sample.
No - this is nonsense, sorry! You must first specify what aspect of the population you are asking the data about. What hypotheses do you wish to contrast? Without a specific question asked, no statistical formula is right or wrong, good or bad. The statistician cannot help you until a specific 'scientific' question has been asked. - - The same was the problem with Anthony's original query: he wanted "some measure of significance between the two methods" - but did not say what difference in the data before him he wanted to declare well vs. poorly ascribable to sampling error.
Dear Ali! Please study what I wrote in my previous remarks. It is essential to specify what question you are asking of you data. Suppose we have
83 25 108 silico successes
17 1475 1492 silico failures
100 1500
maybe this paper and corresponding R-package helps? https://academic.oup.com/biostatistics/advance-article-abstract/doi/10.1093/biostatistics/kxy017/5032579
- Badges
- Science topic
- Similar topics
- Chemistry
- Pharmacology
- Pharmaceuticals
- Drugs