If one is estimating a proportion, x/n, e.g., the number of "successes", x, in a number of trials, n, using the estimate, p.est=x/n, one formula for an estimate of the standard error is sqrt(p.est*(1-p.est)/n). This "behaves well" in large enough samples but for small samples may be unsatisfying. For instance, it equals zero if the proportion is zero. There are a number of alternatives which resolve this problem, such as using SE=sqrt(p.h*(1-p.h)/(n+1)) where p.h=(x+1/2)/(n+1). You might gain some insights by looking at

http://en.wikipedia.org/wiki/Binomial_proportion_confidence_interval

SD = NPQ or Variance = NPQ???

Ops, you're right, thanks!

If one is estimating a proportion, x/n, e.g., the number of "successes", x, in a number of trials, n, using the estimate, p.est=x/n, one formula for an estimate of the standard error is sqrt(p.est*(1-p.est)/n). This "behaves well" in large enough samples but for small samples may be unsatisfying. For instance, it equals zero if the proportion is zero. There are a number of alternatives which resolve this problem, such as using SE=sqrt(p.h*(1-p.h)/(n+1)) where p.h=(x+1/2)/(n+1). You might gain some insights by looking at

http://en.wikipedia.org/wiki/Binomial_proportion_confidence_interval

In Binomial distribution, Variance=n*p*q, therefore SE=sqrt(Variance/n)=sqrt(p*q). Is it correct? Why do you say SE=sqrt(p*q/n)?

Both the n's are different! n in variance refers to number of trials and n in SE refers to sampling!!! I guess if two different notations were used, then it would be clear!

I think probability of finding a pathogen might follow Poisson distribution well than binomial. have you tested the distribution of your data? For a Poisson distrn you can calculate SD or SE by standard formulae.

Are you sure? Can you tell me the formulas for SD and SE within Poisson and Binomial distributions?

For binomial distribution, SD = square root of (npq), where n= sample size, p= probability of success, and q=1-p.

For Poisson distribution, SD = sqrt of ( lambda), where lambda is the mean number of occurrences of the event in a given time interval.

As you stated, your data pertains to number of pathogens in plant tissues over time, you may use Poisson distrn. If your sample size n is large, say > 30, and probability of finding the pathogen is small, say p< 0.05, then you can use binomial distrn.

And what about SE for both distributions?

Let me explain better my experiment: at each time point, I have evaluated the presence of the pathogen within different tissues of number of plants. Therefore, at each time point, I have a number of successes (say x, pathogen presence) on a number of events (say n, number of isolation attempts). Of course, I have x and n per each time point, tree, and tree organ. Let's say p=x/n (viz. total x/total n), and q=1-p.

Therefore:

Mean = n*p

Varaince = n*p*q

and for Binomial distribution (when Mean>Variance, p=constant)

SD=sqrt(n*p*q)

SE=sqrt(n*p*q)/sqrt(n) ---> is it correct?

and for Poisson distribution (when Mean=Variance, n>30-100, p is it correct?

However, my response variable is a proportion, mean>variance, p is very variabile among samples and on average =0.5, n per sample = 10-40, total n = 2055. Therefore I think that a Binomial distribution, and a logistic regression should be used. Do you agree?

If you do have proportions, then the binomial model is the best. The Poisson model is only a different formulation (as a limitting case of a binomial) where there is no information about the total number of trials available (or not meaningful).

I do see more complications in the design, where several organs per tree are analyzed (-> dependencies between organs within tree). I further do not see if the data of the time points are independent. When the analyses are done at different times on the same tree, then the data are not indipendent. Here you might need a mixed-effects model (for a binomial response) to get sensible estimates for the variability (over time, between organs within tree, between trees [for each organ?]). But I am not an expert here.

At each time point, different tree were used, meaning on February, tree 1,2,3,4, with roots, trunk and branches, then on May, tree 5,6,7,8, with roots, trunk abd branches. Trees were always sampled randomly. Also, I considered the option to use a repeated measure analysis, with time as a repeated measure on the subjects (trees), but indeed this is not the case because the trees at each time point were different.

What is your objective? how many total number of trees you have planned to investigate? this is a bit special design. If you have x and n at each time point, are you going to apply binomial for each time point or for all together as you mentioned average p=0.5 and total n=2055? then what is your hypothesis for testing?

Well, after reading all your comments, and the book 'Statistical distributions 2nd ed.', Wiley (1993), I must modify my last posts and summarize my idea.

Given that:

- x=number of successes

- n=number of trials (that comprises x)

- p=probability of successes

- q=1-p=probability of insuccesses

Bernoulli distribution is just a Binomial distribution with n=1

then, with no samples within a population:

- Mean=np

- Variance=npq

- Standard deviation=sqrt(Variance)=sqrt(npq)

- Standard error=sqrt(Variance/n)=sqrt(pq)

However, this scheme does not agree with the statement of Todd (who said SE=sqrt(pq/n), and with this link http://stats.stackexchange.com/questions/29641/standard-error-for-the-mean-of-a-sample-of-binomial-random-variables.

Who can advice on this scheme compared with his own knowledge and eventually some references?

Shashi, my objective is to calculate standard error for each mean probability in the attached graph, to add standard error bars to them

The key issue to me is that if you count the number of 'positive cases' you need to be careful that each of the tests is independent of any other test. So if you have samples form the same plant - can that be considered as independent? If not, the problem becomes much more complicated.

If you do have independent samples one idea is to use as flags the 95% CI around the incidence. I would recommend the Clopper-Pearson method, that you can use from internet calculators (it is more robust in particular for small numbers and incidences close to 0 or 1). The interpretation is neat in the sense that those bars provide credible region for the 'true' incidence to be (ie it is 95% certain the that region includes the true incidence), assuming your samples are random - and representative.

In order to check formulas for SE, another question has arisen in my mind: given that the Binomial distribution tends to the Normal distribution when n is high (i.e.>30), SE in Binomial distribution and SE in Normal distribution should be equal when n is high. For the purpose, I invite you to take a look at the attached file. It was strange to me to see that, using the above formulas, SE in Binomial distribution corresponds to SD in Normal distribution for any size of n.

Of course, I was wrong in somewhere, but where?

I think there is some confusion what the SE refers to. The SE always refers to an estimate. In the binomial case, the parameter p (or q as q=1-p) is usually estimated from the number of trials (n) and the number of successes (k). The maximum likelihood estimate is p = k/n. This estimate is given with a precision, and this precision may be expressed in terms of the standard error, what is SE(p) = sqrt(pq/n).

Using the normal model, we would need to code "success" as 1 and "fail" as 0. The maximum likelihood for k successes and (n-k) fails is the mean value, that is (k*1 + (n-k)*0) / n = k/n = p, the same estimate as for the binomial error model.

The variance as the average squared deviations is then (kq²+(n-k)p²)/n. For n->Inf the value of k will approach np, so the variance will approach

(npq²+(n-np)p²)/n = (pq²+(1-p)p²) = pq²+qp² = pq(p+q) = pq(p+(1-p)) = pq

the SD will thus approach sqrt(pq) and the SE will approach sqrt(pq/n), what is the same as for the binomial error model.

Wonderful Jochen, this is just what I desired. Sorry for my incompetence in statistics and mathematics :(

And, sorry for my other doubts:

- what's the variance in Binomial distribution, npq or pq?

- if k=pn and n->inf, thus k->inf but not k->np. Where am I wrong?

Giovanni, the mean of the binomial distribution of events is np and the variances is npq. The SD of p is given by sqrt (pq/n). Second question is not clear.

To the second question: for n->Inf, k=np -> Inf, too. So you are not wrong. It is just that one would not recognize the similarity of the variances (and SDs, and SEs) between the two distributions if one would just substitute "k" by "Inf". Note that the equality is approached for n->Inf, where both the SDs and SEMs become zero.

More about question 2. Although in general k does not converge to np as n tends to infinity, it's important that k/n (frequency estimate, a random variable) does stochastically converge to p ("true" frequency, constant parameter of the distribution). The SE of k is actually the SD of k/n, so it is equal to sqrt(pq/n) as noted above.

Shashi, we have said that sqrt(pq/n) is SE, not SD.

Now, it is not clear to me what is the Variance in Binomial distribution. In the books, I found Variance=npq, however by comparing Variance values in Binomial and Normal distributions (as done in the xls file attached above), I realised that Variance=pq in Binomial distribution (this would make clear because SE=sqrt(pq/n)).

If I'm right, why in the books Var=npq, while I realised Var=pq?

in a binomial experiment, the variable of interest is number of successes or positive results. this will be in the form of a sum of Bernoulli experiments which are assumed to be independent and identical. a Bernoulli random variable has variance=pq, hence a binomial random variable will have variance=npq because the variances of the Bernoulli experiments will just be additive.

i wasn't able to follow all discussions in the thread, but i think your interest is not the sum of the successes but the mean or average success (which is sum of independent and identical Bernoulli experiments divided by the number of trials, say n). In which case, the variance of this sample proportion or average success will be pq/n

it should be made clear, i guess, that it is the total number of successes which has the binomial distribution, not the proportion of successes over a fixed number of trials.

Giovanni if (I quote) "The probability in the graph is a mean of several replicates" you should consider to use a replication method to compute your standard errors. In fact in your experiment your "true" degrees of freedoms depends on the replication exercise. In this case you should divide a measure of your standard deviation by a the number of the replicates (or a transformation) and compute your tests (if any) accordingly. As a reference (among many others) see the survey Variance estimation for complex surveys using replication techniques.Rust KF, Rao JN. Stat Methods Med Res. 1996 Sep;5(3):283-310.

Ok, let's have a look at the attached graph.

In 2008, the value in the graph is 20.8%, meaning p=0.208=820/3940. This means that, in 2008, 820 pathogen colonies (successes) were obtained from 3940 isolation attempts (trials).

Is this a binomial experiment, viz. a sum of Bernoulli trials? And what are SD and SE here?

I feel that the problem here is that you want statistics but the purpose is not clear. What is the standard deviation of a proportion? What information would it convey to a reader? And likewise the standard error?

Why don't you graph confidence intervals for your proportions. This gives the reader a useful idea of the precision with which you have measured the underlying parameter. (The standard error does not, since its interpretation is a 66% confidence interval!).

Note that the textbook formula for the standard error of a proportion is a hopeless approximation. If we use it to construct the confidence interval for an observed proportion of 2 occurrences in a sample of 23, the SE is 0.55 and the 95% confidence interval is -0.03 to 0.19. Clearly this is nonsense. The proportion cannot be zero, since we have observed two successes, and no proportion can be less than zero, so the confidence interval includes values that are inconsistent with the data (zero) and undefined (negative).

SPSS, by the way, gives these nonsense confidence intervals with a straight face. For that much money, you have a right to expect something a lot better.

In case you wonder, the general advice is to use the Agresti-Coull confidence interval for N > 100 and the Wilson or Jeffrey's interval (they are equivalent) for N < 100. At N=100 the results are identical – I checked!

Sorry if my question seems more complicated than it is.

My purpose is to present my data with a graph (as the attached one), and this graph should report means and some parameter (SD, SE or something else) to give idea to the reader about the precision/variability of the means. Of course, this graph will be included in an article together with several others. Also, since the graphs will report several kinds of data I would prefer to use the same precision/variability parameter for all the graphs (=kind of variables), and by the way I was thinking about SD or SE. That's all.

if you think of each isolation attempt as trial, the presence of pathogen colony as success with constant probability from trial to trial, and the trials are made independently, then the value you obtained for the sum of Bernoulli trials is 820. the value 820/3940 is the proportion of success. as explained earlier, the sum of Bernoulli trials is the one with the variance of npq (p in your experiment is unknown). the value 820/3940 is only an estimate of the value of p.

If you're presenting averages of different replicates per each year, the standard deviation of those averages (among different replicates) is an estimate of the standard error of the parameter you're estimating in the population. The problem, as I stated in my previous post, is that the std err formula should change according to the replication method adopted (balanced repeated replication, jacknife or bootstrap).

First I would like to understand SD and SE in a simple case, viz. 3940 attempts (independent trials). Then I can ask for cases including replicates.

Thus, what are SD and SE in this particular case?

Let's imagine that for 2008 your 23.1% average is the result of the mean of 10 different replicates (each observation is the average of a subsample of your data):

18 19 19 20 20 25 26 27 28 29.

Imagine that the sample size of each replicate is the same of total sample size, that each replicate is sampled with replacement and that you do not have info on the population.

Now the std deviaton among those replicate is an estimate of the std error of your mean (23.1%): in my simple example this is 4.3%, and a normal (approximated) confidence interval of your estimates with the 95% confidence interval should be include between (23.1-1.96*4.3) and (23.1+1.96*4.3)

The SE is DE/M; where DE= SQRT(SUM(p_i(1-p_i) )) where p_i = Success/n for the replicate i. n = sample size for each trial and M= number of trials. Why? because the the estimator is the mean of sample of M proportions;

You can compute SD and SE for the probability of occurrence of a binary variable according to the association model that you have already defined. In order to do this in SPSS, after defining the regression model, you can save the probabilities (you may tick the option in the model dialogue box) and after running the model, you can calculate SD and SE for the computed probabilities that has been saved as a new variable in your file.

Felipe,

SE=DE/M or SE=DE/sqrt(M)? and DE=sqrt(SUM(p_i*q_i) or DE=sqrt(AVERAGE(p_i*q_i)?

Yes, SE = SQRT ( SUM (p_i*q_i) /n ) / M; the SUM is for i=1 to M. I forgot to divide by n

Dear Giovanni Bubici and folowers of the question,

I was following the answers to your question on standard deviation of a proportion. You state a simple question but the noise is considerable. This is not rare, "the simpler is the question, the more difficult or more controversial is the answer". As I am involved in compositional data analysis, I pay attention to most discussions on proportions. In your case, I think that for answering your question there is no need of a compositional answer but is near to. I'll try to answer in three steps, being the first one practical and straightforward.

Step 1. I agree with Ronan Conroy that what you are looking for is not the standard deviation of a proportion, but a confidence interval on it. R. Conroy suggested two methods to give such an interval. However, I would point out that an exact confidence interval for a proportion from binomial events is available: the old, but not well-known, Neyman geometrical method. An (1-a) confidence interval (p_1,p_2), symmetrical in probability, about the probability p, is obtained by solving the equations in p_1, p_2,

F_X( x_o | p_2, n) = a/2 , F_X( x_o | p_1) = 1 - a/2 ,

where X is the n-trial binomial variate; F_X(x|p,n) is the cumulative binomial distribution function for n trials and probability p; x_o is the observed number of counts; and p_1, p_2 are the end-points of the (1-a)-interval you look for.

This approach can be used even if the observed count is x_o=0. However, this puts forward another challenging question: when x_o=0 the maximum likelihood estimate of p is just x_o/n=0, which is outside the symmetrical confidence interval. In fact, what is wrong is not the confidence interval but the estimator. The asymptotic consistency of x_o/n when estimating a small p is only attained after a very large number of trials. I would recommend to use some Jeffreys-type estimator like p approx (x_o + 0.5)/n. However, this estimator can be as disastrous as the traditional x_o/n.

Step 2. This is only a comment on the variance or standard deviation of a binomial. Traditionally, the number of events of a binomial is considered embedded in the real numbers. Then, the distance between a zero count and 1 count is equal to the distance between 100 and 101 counts. In this conditions, as commented in other answers to the question,

E[X]=np ; Var[X]=np(1-p) .

This very straightforward, and apparently sound answer, can collapse when computing intervals using standard deviations (see example by R. Conroy). However, the origin of apparent paradoxical results comes from the fact of considering the absolute scale for the number of counts. If the scale on the counts is changed, both the mean and variance change accordingly (the theory is due to Frechet for metric sample spaces, and is used systematically in compositional data analysis). This leads us to have some doubts about the relevance of the standard deviation of a binomial.

Step 3. Looking at your figure with proportions estimated for only three years, it suggests to me the possibility of using logistic-regression. This approach largely exceeds your problem, but can be useful when the number of years is about 10 or more. Then, you can model the trend on the proportion of successes and give intervals for the whole trend. To me, the interesting point is that what is then estimated is not the proportion p(t), as a function of time t, but the log-ratio (or logit) log[p(t)/(1-p(t))]. This is a common feature in compositional data analysis.

I apologise for this long exposition. I did this to confirm the starting sentence "the simpler is the question, the more difficult or more controversial is the answer".

Dear Juan,

many many thanks for your detailed exposition. However, something you said is difficult to understand for non-statistician people like me. Nevertheless, I realised that the use of confidence intervals may be appropriate for my purpose. Moreover, to analyze my data, I used logistic regression indeed, while means comparisons were made by contrast analysis.

Now, It remains to be defined for me how to graph my data. May you be so kind to do a numerical example (see my data in previous post) to determine condifence intervals and/or log-ratio?

Dear Giovanni,

I think your figure is OK if you substitute the bars by a confidence interval. Then you only need to compute the confidence interval. The estimated proportion can be the standard one k/n or the Jeffreys approach (k+0.5)/(n+1) (as commented by Todd Mackenzie). I attach an excel spread sheet to manually compute a 0.95 confidence interval. Obviously, there are more efficent procedures.

Juan,

thanks for the file. I have not understood how you calculated the 95%CI. Why did you choose just those values in the p list?

I would caution against the so-called 'exact' confidence intervals for a proportion. They are not what you think they are! They do not have exactly 95% coverage for all sample sizes and all observed frequencies. My recommendations were based on : Brown L, Cai T, DasGupta A. Interval estimation for a binomial proportion. Statistical Science. 2001;16(2):101–17. They evaluate confidence interval formulas for coverage based on the problem known as 'lucky and unlucky N and p". I recommend it to anyone seriously interested in this rather tricky problem.

Thanks Ronán for your comment.

In the article you suggested, CI=p±k*(n^-0.5)*[(pq)^0.5]. I cannot understand what k is. Can you explain it?

They explain it as z[subscript alpha/2] or the inverse normal distribution corresponding to (1-alpha)/2. So it's the term that defines the coverage of the interval.

Therefore, for 95%, z=0.025?

Try Matlab, it will give you standard deviations and errors for binary data; command - std2(A); where A is the vector of input

Hello everyone, this is indeed an interesting discussion. I face the exact same problem, though after reading this I am wondering if CI for sample proportion can still be calculated for time-correlated data. In Giovanni's experiments, the observations in each year are independent; however, are the point estimates in each of the three year independent as well? If not, would this pose a problem in CI calculations? 

Hi all,

I am starting analysis of data on R software. I am interested to compare the prevalence of binomial data (0 and 1) between and within different species to make bar with 95% CI. Could anybody suggest me the way how I start with!