I can suggest you something. Do the logit transformation to each porportion first.

logit(p_i)=log[ (p_i) / (1-p_i)]

and logit(q_i)=log[ (q_i) / (1-q_i)] for each proportion of the (p_i,q_i) pairs.

Then do a paired samples t-test on the transformed data.

That is a good suggestion, and I'll try doing that.

My concern is that the variance associated with each p value is p(1-p)/n where n is the sample size. The paired proportions with the underlying sample size are like (2/10,7/30),(1/10,2/10),(4/10,11/30) etc, sometimes n=10 somtimes n=30 for the second paired value. The sample sizes can be different for the proportions calculated. Shouldn't the effect on the variance of these different sample sizes be incorporated in a proper test?

Well, that is true, but you want the variance of the one sample and the variance of the second sample. If you had euclidean data you would do a paired samples t-test. That's what I am telling you to do now. Logit the two samples and do a t-test.

I have a question though. How can they be dependent and of different sample size? IS each pair of proportions within the same cluster?

Michail, Thanks for your suggestion and comments. The proportions are paired because we are testing the toxicity of 2 kinds of fish in a common sample of water. The lab that does the testing adds 10 fishA and either 10 or 30 fishB (I don't know why it changes) to the water sample and count the # of each kind that die to get the 2 paired proportions. Next day we get a new water sample and repeat the testing and so on for a number of days. We want to test if the proportions are the same. In the general case all the sample sizes could be different.

I've tried to find a reference that discusses this hypothesis testing situation with paired proportions with no luck.

OK, I see. The problem with the logit is that you do not actually compare the means though, but rather their medians. Aitchison suggested this transformation in 1980, 1982 for multivariate proportions. About the normal approximation to the proportions I do not think it is directly applicable here. I will also have a fast look now at google scholar and let you know.

I did not find anything. It's strange though that nobody has thought of anything.

Or maybe has, published it somewhere where google scholar does not show. Alternatively you can use the non parametric wilcoxon test. That does not understand the scale of the data, only their sign. Maybe it's caled sign test then. But I suggest if you do this, use bootstrap.

Michail, thanks for your interest. If you look at the test for comparing 2 proportions you see how the sample sizes are involved. You could test the proportions 1 pair at a time, but that can't be the best approach if you have a data set of paired proportions.I would think this data analysis problem must be common, but I couldn't find it discussed anywhere.

Yes, I agree with you. In the worst case scenario you can do that. All the paired proportions tests and compare their p-values against 0.05/(number of pairs). But this will give you many p-values, not one. But wait, yes then you know what. In this case you can do a meta-alaysis. You have all the 20 say p-values, based on 20 tests.

1) sum(log(p-values)) follows a chi-square I think.

2) A combined estimate of the z-test, either fixed estimates or random estimates. This will be your final answer.

How about this?

It is interesting that it is so hard to find a reference for this type of problem. You would think it would be fairly common, and somebody would have studied it long ago, and it would be discussed in all the usual reference books.

Your suggestion is very interesting. Are you sure the sum has a chi-sq disribution, and how many degrees of freedom would be used?

Unless it is published in a not so popular or mainstream statistical journal.

It's actually one of the many Fisher's tests.

http://en.wikipedia.org/wiki/Fisher's_method

As for the second thing, the combined estimate, see meta-analysis for that for more information. There is a package in R which does metanalysis also. You just feed the estimates and their variances and it does random and fixed effects. A test tells you whether random effects are significant.

So you would use the differences and the estimated variances of those differences?

Yes, that is true.

Apologies for the delayed response.

I believe the optimal approach here might be to work with loglinear models (the extension of contingency tables to the case with more than 2 factors).

Rather than working with proportions, you work with frequencies. Basically, you have three factors:

A: Fish Type (either A or B)

B: Survival (either yes or no)

C: Replication (day number)

Suppose on day 1 you had proportions (2/10, 7/30); day 2 had (1/10, 2/10), etc. (like in your example above). Then, the data structure would be as follows:

A B C Freq.

1 1 1 2

1 0 1 8

2 1 1 7

2 0 1 23

1 1 2 1

1 0 2 9

2 1 2 2

2 0 2 8

etc.

Next, you fit a straightforward loglinear model (see https://en.wikipedia.org/wiki/Loglinear_analysis and the references therein for the basis of this method) and you can test whether there is a 'factor A effect' (= whether fish type A differs from B), but also whether there is a replication effect. Good luck.

Casper, thanks for your suggestion.It sounds very promising. I'll give it a try. Bruce

Hello Bruce, I've run into the same problem. Did a loglinear model work for your data? Thanks.

Rochelle, The loglinear model did seem to work. Its an interesting problem and it seems to me that it should be fairly common but its impossible to find a reference anywhere. I hope this and the other comments help you. Regards, Bruce Wilson

Hello, I am tasked in calculating the power of my sample size for detecting in changes in proportions (paired proportions) of bugs with a certain drugs before and after treatment. Since I do not know the standard deviation of the proportion i must simulate and make a graph of standard deviation and effect.I know the sample size is 150, alpha 0.05 and power 0.8

What is the best way to do this??

Michail Tsagris