Jochen, what you are describing sounds appropriate for comparing two observed proportions. But the way I read it, 1/10 is the expected proportion according to a null hypothesis. If one was going to perform a test rather than compute a CI, I think it would be a binomial test, with H0 specifying that Pi = 0.1. Using Stata's -bitesti- command, I get this result for the 60/100 example that was given (copy to text editor and view with fixed font to make the output align properly):

. bitesti 100 60 0.1

N Observed k Expected k Assumed p Observed p

------------------------------------------------------------

100 60 10 0.10000 0.60000

Pr(k >= 60) = 0.000000 (one-sided test)

Pr(k = 60) = 0.000000 (two-sided test)

note: lower tail of two-sided p-value is empty

Cheers,

Bruce

The simplest way to get a confidence intervals for the difference is, I think, to use a generalized linear model of the binomial family. The pairing structure can be accounted for by a fixed or a random effect for the subject.

You second question ("how certain I am in this ...") requires a Bayesian analysis. This would be answered, I think, by the posterior odds.

Dear Researcher

Most of Statistics books were studying and calculating CI.

Regards,

Zuhair

...and most of these books only show CIs for nomal distributed variables, and even then the concept is often only shown for getting a CI for a mean, not for a mean difference (or more general for any coefficient in a linear model).

Jochen, I read the question a bit differently than you, I think. Rather than a CI for a (mean) difference, I think Jan is just asking for a CI for the binomial proportion--e.g., 60/100 in the example he gave. I think he wants to know whether that CI includes or excludes a value of 10%.

Jan, if that is what you are looking for, I would recommend using the Wilson-score method to compute the CI for the proportion.

Newcombe, Robert G. "Two-Sided Confidence Intervals for the Single Proportion: Comparison of Seven Methods," Statistics in Medicine, 17, 857-872 (1998).

For the example you gave, the Wilson 95% CI is: .5020026 to .6905987

I computed that with Stata's -cii- command. You haven't said what software you use. If it does not have a canned routine to compute it, the code is not that difficult. Here is some SPSS code you could adapt if necessary. In the code, z = the critical value of z (e.g., 1.96 for a 95% CI) and ** means "to the power of"--e.g., z**2 = z-squared.

* Wilson score method (Method 3 in Newcombe, 1998) .

* Code adapted from Robert Newcombe's code posted here:

http://archive.uwcm.ac.uk/uwcm/ms/Robert2.html .

COMPUTE #x1 = 2*n*p+z**2 .

COMPUTE #x2 = z*(z**2+4*n*p*(1-p))**0.5 .

COMPUTE #x3 = 2*(n+z**2) .

COMPUTE lower4 = (#x1 - #x2) / #x3 .

COMPUTE upper4 = (#x1 + #x2) / #x3 .

There are also online calculators, like this one:

• http://vassarstats.net/prop1.html

HTH. Sorry if I've misunderstood what you're trying to do.

Bruce,

it might be that this was what Jan had in mind when he asked the question. But the actual question is really for the mean difference. It is a common (bad!) practice that the significance of a difference (e.g. between 10/100 on 60/100) is evaluated by looking at the CIs of the two means, and if they overlap. This is, I think, a consequence of how CIs are taught in books (see my comment above). If the question is: "whether the performance of this participant over a series of trials is better or equal to chance" (where "chance" was defined as 10/100 in the sentence before, then it is clearly about a change, about the mean difference between 10/100 and 60/100, say. It's neither about 10/100 not about 60/100. For me that seems quite clear from the question. Let's hope Jan will enlighten us ;)

Jochen, what you are describing sounds appropriate for comparing two observed proportions. But the way I read it, 1/10 is the expected proportion according to a null hypothesis. If one was going to perform a test rather than compute a CI, I think it would be a binomial test, with H0 specifying that Pi = 0.1. Using Stata's -bitesti- command, I get this result for the 60/100 example that was given (copy to text editor and view with fixed font to make the output align properly):

. bitesti 100 60 0.1

N Observed k Expected k Assumed p Observed p

------------------------------------------------------------

100 60 10 0.10000 0.60000

Pr(k >= 60) = 0.000000 (one-sided test)

Pr(k = 60) = 0.000000 (two-sided test)

note: lower tail of two-sided p-value is empty

Cheers,

Bruce

Bruce, yes, that's what I thought: the 10/100 wasn't a theroetically justified value but the result of an observation in a control group. Your view makes very much sense when the task was, for instance, to select one out of 10 answers in a multiple-choice test, where one answer is correct, and where the participant would blindly select one of the answers (and this is repeated 100 times). Good that you pointed this out!

It still would be nice if Jan clarified this.

To make my question a bit more clear:

Assume a participant is conducting a task where s/he has a 1/10 probability of being correct by chance. Assume the participant has 60% successes over 100 trials. At no point I want to use group-level data. The 10% chance of being correct is simply a function of the task it self. (So, much like how Bruce read my poorly formulated question).

Testing whether the performance of this participant over a series of trials is better than chance could be done by calculating the cumulative binomial probability of .6 being larger than .1 in a simple binomial test. This, I assume, provides a probabilistic answer to how certain I can be this participant being better than chance. Conversely, the odds of it being a fluke should be 1 - this probability.

But, I also want to say something about precision (e.g., CI for this estimate) and use this as a lower and upper bound prediction of the performance I expect from the same participant in a new set of similar trials.

Why not go bayesian,

with priors Beta(p,alpha,beta) with a set of

alpha = 0.1 , 1 , 10 and beta = 9*alpha

being more and more precisely around 0.1 for the probability of success.

If a student hit a score of 60 amongst 100 items the priors are

Beta(p,alpha+60,beta+40)

densities shown on the attached fig.

and posterior Pr(probability of success < 0.1) all below 1e-20

if they are more than 1 student, you may be more subtle (with something related to Rasch model e.g.)

Bruce seems to right on the mark.