I have a dataset where 56 subjects are compared to a reference population norm for height. I evaluated subjects on whether they are above or below the 10th percentile of this reference population norm and created a variable coded for each subject as either a zero (above the 10th percentile) or 1 (below the 10th percentile). This variable has a mean of 0.0357 (95% CI=.01 to .08)---so about 4%, or 2 subjects, were below the 10th percentile. To determine statistical significance (i.e., is this group of subjects significantly more likely to be below the 10th percentile of the reference population than is expected), I used a one-sample t-test comparing the mean of the variable to a hypothesized mean of .10 and got a p-value of .013. This seems to make sense because the 95% CI of the mean excluded .10. However when I ran a nonparametric binomial test on this same variable, comparing it to a hypothesized binary probability of .10, I got a p-value of .084. Which test should I use?
My intention was to create a variable so I could compare the mean likelihood of these subjects being below the 10th percentile of the population to what the reference population's likelihood is, which is of course 10%. I don't have an N for the reference population; I just knew what their 10th percentile cutoff value was for height.
CALCULATIONS: Suggest to reconsider in light of comments below.
BINOMIAL: You define below population mean as success (s = 1) and else 0. You counted s = 2. The Laplace rule of succession follows:
(1) p = (s + 1) / (n + 2)
(2) q = 1 - p
Thus, q = 1 -.05 = 0.95. You asked whether the 2 counts is statistically significant? The test statistic for binomial distribution follows:
(3) Z = [(x / n) - p] / SQRT(pq/n)
Z = [(2 / 56) - 0.05] / SQRT((0.05/0.95) / 56)
Z = (0.04 - 0.05) / SQRT(0.0008)
Z = -0.01 / 0.03
Z = -0.33
From the negative Z-Table, Z(obs = -0.33) has a critical value of 0.3632 or probability of 36.32% chance. This is not statistically significant.
T-TEST: Assume that you compare the 2 counts of heights below the mean to the population mean, and ask whether the difference is statistically significant?
(4) T = (X^ - y) / (S / SQRT(n))
... where X^ = mean of the two counts; y = population mean; S = sample standard deviation (i.e. standard deviation of the two counts) and n = 2 not 56 (because you are seeking to verify those 2 counts not the sample of 56). The T-critical here is about T(0.95) = 1.68 for df = 50.
ALTERNATIVE STANDARD SCORE: Your objective is to verify whether the two counts below the mean is statistically significant (by comparing to the population mean (not the mean among 56 counts). The standard score than is:
(5) Z = (Xi - y) / S
... where Xi = each of the 2 counts you found below mean; y = population mean; and S = standard deviation of the 2 counts. For Z(0.95) the critical value is 1.65. If any one of the 2 counts shows Z(obs) > 1.65, then it is significant if beloe 1.65 then it is not significant. This approach is the simplest.
WHICH METHOD TO USE? The binomial method answers the question of whether the success counts in relations to the total sample size is significant---it does not answer the question of whether a specific height in relations to the population mean is significant. The Z-standard score answer whether a specific height in relations to the population mean is statistically significant, i.e. significantly low or high.
REFERENCES: see attached files.
Binomial test.
The T-test does not apply here since a binary variable is obviously not Gaussian so the T distribution does not apply (so type I error is not known), and the variance estimation used in the denominator should lead to a much less powerful test than using the formula for the binomial distribution.
Note that T-test at the 5 % level and 95% CI of the mean not including your reference value are exactly the same thing, so they cannot give different results; note also that, consequently, here your CI on the « mean » is wrong if computed as for a mean and not as for a proportion.
An example of the binomial test and the code for SAS can be found here:
http://www.biostathandbook.com/exactgof.html
Code for R for those examples can be found here:
http://rcompanion.org/rcompanion/b_01.html
Each will give you a confidence interval about the proportion as well.
Thank you all very much for those answers. I suspected that using the T-test wasn't quite right, although when I created that binary variable it was enticing to use the T-test at first because the variable had a mean and a standard deviation.
- Badges
- Science topic