Determining intervals for the common language effect size (CLES), probability of superiority (PS), Area Under the Curve (AUC) or Exceedance Probability (EP) is possible via multiple method Ruscia and Mullen (2012). However, is this also possible via Fishers Z transformation? For simplicity I will call the “effect size” EP.
If we make the following assumptions: we have a (real) value that can range between -1 and 1 and assume the error distribution is (approximately) normally distributed (also invoking CLT), then we would be able to obtain intervals via Fishers z transformation (I think???).
The rationale is: EP does not range from -1 and 1, but from 0 till 1. Hereby 0.5 would represent the NULL. However, it would be possible to transform the EP to a value between -1 and 1 assuming a “directionality”: < 0.5 is negative and >= 0.5 is positive. Then,
EPd =(EP-0.5)*2
EPz = ln[ (EPd+1)/(EPd-1) ]*0.5 = atanh(EPd)
SE = √[ 1/(3-n1) + 1/(3-n2) ]
Lower = SE*1.96-EPd
Upper = SE*1.96+EPd
Transformation back to the original scale (EP) would be possible for both positive and negative values.
If positive
EP = [ exp(EPd*2)-1)/(exp(EPd*2)+1 ]/2+0.5 = tanh(EPd)/2+0.5
If negative
EP = [ 1 + (exp(EPd*2)-1)/(exp(EPd*2)+1) ] / 2 = [ 1 + tanh(EPd) ]/2
However, when comparing the analytical intervals to Monte Carlo (MC) simulations the intervals are much broader using a smaller samples size. Although the extreme intervals, either upper when EP < 0.5 or lower when EP < 0.5 Below an example of Ruscio and Mullen (2012) where n1 and n2 are both 15 and another example with the same mu and sd when nx and ny are both 150. Also the intervals by Ruscio and Mullen (2012) are much smaller. The question is, why are these intervals broader is the rationale completely wrong, did I make a mistake or is it simply impossible what I am doing? I know there are other ways obtaining the intervals but using fishers Z transformation would make it rather “elegant”.
Thank you in advance for your time!
Hello Wim,
I think your scenario doesn't match well to the situation for which Fisher initially developed the z-transformation (r-to-z). The issue with the distribution of the Pearson correlation (r) is that it does not have a constant variance and the r values don't behave linearly very well (unlike r-squared). So, the z-transform is principally a variance-stabilizing measure, which does tend to yield a z-variate that conforms better to the normal distribution than does r.
But, you're presuming that the variate to be transformed has an underlying normal distribution; that's unlike the r-to-z situation. As well, I'm guessing that your intention is to convert some proportion of area (of non-overlap) to a z-score or some other standard score (not the z-transformation, above), which is also presuming a normal distribution for whatever attribute is under investigation, which may not be the case (and frequently is not in practice). Why not stick with proportions as your explanatory ES, as in:
1. CLES of McGraw & Wong: Proportion of times randomly selected case from batch A has score that exceeds that of randomly selected case from batch B?
2. Cliff's dominance statistic: CLES - proportion of times that B cases exceed A cases? Article Dominance Statistics: A Simulation Study on the d Statistic
3. Vargha & Delaney's A statistric (similar to Cliff's d, though it also splits tied cases and assigns half to one batch and half to the other).
Each of these is exact. There is a pretty good closed form expression for the variance of d, if this is important.
Good luck with your work.
I think the key question here is what is the theoretical probability distribution of CLES or EP, provided that the data come from two independent normal distributions. Numerical examples of Monte Carlo simulations may help answer this question. Just a thought.
Hello David Morse thank you for your response indeed if the underlying distributions of the two groups is not normal this might influence the results and an exact solution would be preferred. However, I noticed this only "marginally" impacts the results. Herby marginal is left vague as this a vague term in general. I was just curious if it would exist. The use of the binomial error is easiest and sqrt(pq/n) gives reasonable results in comparison the MC. The error could then simply be the input to beta distribution approximating the 95% intervals (as alpha=ep*n and beta=(1-ep*n)), which as a difficulty excluded 0 or 1 exactly. In any case, just curious.
Hening Huang thank you for your response never really found the time to check the performance of EP between rank (via U statistic) and the assumption of normality (Wong). However, I found some clarity in the use of binomial error in this case and the help of the comments in the other threads.
Some examples, given we have have two uniform populations U(0,100) and randomly generate samples from from each population in a sequential increasing order 5 till 150 and repeat this 50 times (The sample sizes are always equal). If I apply the rank and normal method of EP over and under estimation is ~equally valid for both methods. Even though the ranked method would give an exact solution the distributional aspects of the output seem not so different (Fig EP_U(0,100)_2x). However, when using LN(4.3, 0.6) and U(40, 130) the normal method seems to work better when sample size is smaller ~
- Badges
- Science topic
- Similar topics
- Mathematics
- Statistics
- Sampling Design
- Sample Size