Sal Mangiafico Yes, you are right, I will polish the question a bit.

Dear Wim,

I have no straight forward solution, but my naive approach would be just to calculate the CLES from the posterior distribution of the MCMC chains. You could calculate the CLES (or any other ES, like Cohens Delta) from the posterior parameters of every step of the MCMC and then analyze the distribution of the CLES posterior distribution, e.g. 2.5 and 97.5 percentiles (or whichever boundaries you prefer), the Mode or Mean of the distribution and even Credible Intervals such as HDI.

With this approach you dont need to impose any priors to the CLES, but only to the data and model parameters, the CLES then "naturally" occurs in the posterior/MCMC results. How do you think about it? Suggestions from others?

(See Kruschke - Doing Bayesian Data Analysis or his BEST package, where he does exactly this with Cohens delta)

Best,

Rainer

Rainer Duesing I will give this a look. I went through the book searching for ranked statistics, but it was mentioned the book will not go in to this. Some sources were mentioned though, still need to look this up.

I am not sure how to impose some parameters on the likelihood functions of the data, this is somehow what you do not really need to be concerned with for ranked statistics (the beauty of ranks as well as its ugliness). This is where the Dirichlet distribution comes in, as then this is avoided (although not very elegant).

I am not sure about your approach and for what you need any rank based things, but please correct me, if I understood something wrong. As far as I know the CLES (McGraw & Wong, 1992) it is the probability associated with a z-score calculated from the group means and the group standard deviations, correct? The posterirors of the MCMC will provide you with all four information. Therefore, you can simply calculate for each sample of the posterior (e.g. 10,000) the CLES from the posterior parameters.

Therefore, I do not understand for what you need the prior on the ranks and Dirichlet distribution in the first place.

Rainer Duesing You are correct that it is possible via the method you propose, I will try this one as well. Please give some thought on the examples below. As I think you had in mind:

cn

Dear Wim,

I still do not understand why you impose the ranks and calculate then your version of the CLES, can you provide a source? Your approach seems to be more a bootstrapping than a Bayesian approach, but you directly asked for credible intervals, didn't you.

Please find a version of the robust t-Test using a student t-distribution like Kruschke to catch very heavy tails in the distribution and the possibility for unequal variances. Maybe not optimal, but in the Bayesian framework you could try other distributions. I used a translation of Kruschke's approach into the "brms" package as proposed by Solomon Kurz.

I ran the bayesian model on your simulated data (nice random seed....;-) ) and used the posterior distribution to calculate the CLES. The results of the point estimates (median or mode) are very similar to the traditional approach, but you can also inspect the posterior distriution for CLES and calculate the HDI. Hope it gets clear with the code:

library(brms)

library(tidyverse)

library(tidybayes)

library(broom.mixed)

library(HDInterval)

###########################################################using your data x and y

lx

Your second rank based approach seems to be indentical in the results to the "empirical distribution" approach from the "canprot" package to calculate CLES, but I cannot find any sources for the rationale, MacGraw and Wong only discuss deviation, but do not show their calculations.

But if you look at the boxplots from all three approaches (Bayesian and your two bootstrapping), the Bayesian robust distribution and unequal variances approach seems to have results somewhere in the middle of the normal and emirical approach.

Not sure about the rationale, but I found the articles below very useful.

I also get an error :(

Error: 'makevars_user' is not an exported object from 'namespace:withr'

In addition: Warning message:

In system(paste(CXX, ARGS), ignore.stdout = TRUE, ignore.stderr = TRUE) :

'-E' not found

I will polish the Frankenstein CLES a bit an drop the code online. I would like to create some wrapper function with all the different possibilities, then I could run it on some simulated and available data.

Article Confidence Intervals for the Probability of Superiority Effe...

Article A Probability-Based Measure of Effect Size: Robustness to Ba...

I found this regarding the error message, maybe you need to update / reinstall "withr"

https://discourse.mc-stan.org/t/why-shows-error-makevars-user-is-not-an-exported-object-from-namespace-withr/17357/5

I will have a look at the papers, thx

This was the idea (see script). I was not able to resolve the issue by reinstalling. I will check by restarting or else fumble it into jags. I tried six different approaches:

1) VDA rcompanion (bootstrap), 2) "empirical" or "rank" approach (bootstrap), 2) the "McGraw and Wong" or "normal" approach (bootstrap), the rank approach with confidence intervals according to Hanley and McNeil [1], Frankenstein rank approach (as given in the script) and the same Frankenstein approach, but based on the "normal" approach. For 2-3 I selected the mean of the bootstrap, but should be the mean of approach 4. Approach 5 is in the script and 6 is not in the script (for simplicity). 1-4 are similar and 5 clearly pops out and 6 is more "moderate". I still want to include the MCMC on "normal" but, did not found really the time, yet. In the script you only need to set some "prior" assumption on the expectation on CLES (AUC), but not on the data (not sure if this makes any sense though).

[1] Article The Meaning and Use of the Area Under a Receiver Operating C...

A small remark: Common Language Effect Size (CLES) is not "effect size". Like some terms in statistics, Common Language Effect Size is an inappropriate term. For two independent random variables X1 and X2, the effect size under consideration is X2-X1 (or X1-X2). CLES is approximately the probability that (X2-X2)>0, or Pr(X2>X1), i.e. the exceedance probability (EP): Preprint Exceedance probability analysis: a practical and effective a...

Hening Huang Yes, I am a bit confused about the term effect-size in this case as it has no unit. But it is also not really a probability as it is a proportion or fraction in the data. Or it is a probability on the data? Not sure anymore.

Also the term effect-size might be appropriate under experimental setup, as the term effect implies a causal meaning, under field conditions difference-size estimator might be a better term (not as fancy). Yet, I am not really sure why you would focus on an "overlapping" effect-size if you have very controlled conditions. Under controlled conditions it might be more reasonable to focus on the mean difference (or otherwise) under the assumption the noise is only cause by i.e. biological irregularities. I simply use it because I have no controlled conditions an everything is noise. But it is useful in somehow seeing if groups of species are very distinctly positioned along a gradient and if this more-or-less similar in other datasets.

I read your article, I like it, I send you a pm on some questions i.e. what is c4 and how to calculate it and if you have some example code?

Wim Kaijser Since CLES is calculated using the z score, it is a probability (analogy to p value). Indeed, it is a frequency if it is directly calculated as a fraction in the data.

I don't quite understand your comments related to "overlapping" effect size and "controlled condition". Mean difference (or mean effect size) is always a useful statistic and should be reported. But EP analysis, especially EPa(0) (similar to CLES) provides more insight into the data.

I received your message and replied with two attachments the day before yesterday. Apparently the message and attachments were not through. I just resent. Please check.

Hening Huang Yes I received it, thank you.

I think overlapping might be some sort of ecology jargon. Where the niche of two species might be very similar to each other, thus "overlap" completely i.e., CLES = ~0.5.

With controlled I mean that you could perform an experiment under laboratory conditions of species A and B keeping everything constant except for three treatments. For example, the growth rate of species A and B under different phosphorus concentrations. The phosphorus concentrations in the three treatments are very similar and noise control is very high. In my opinion it is reasonable to use the difference between the means as an effect size in this context as it gives clear information. If the question would be what is the mean difference in growth rate of species A and B und different concentrations of phosphorus.

The "complete opposite" would be to use a random obtained dataset, containing observations of occurrences (not growth rate) from species A and B in the field under different phosphorus concentrations. Here there is no control and the noise is likely very high affected by other variables/factors. Hence, it might be worthwhile to see if species A and B are very "discriminatively" positioned along the phosphorus gradient. This would be more or less the "realized niche". In this sense the question could be, how often exceeds the occurrence of species A that of species B along the phosphorus gradient.

The use of the answer resulting from to the latter question might also be more pragmatic (less experimental an causative). Lets say we obtained a CLES of 0.67 P(A > B). If I go outside and observe species A, I could get a hunch that its is 2 times more likely that phosphorus is "high". Yet, we do not know how high, only that species A indicates that it is likely higher than when we observed species B. If CLES was 0.55, I would not consider it very worthwhile to think of this distinction. If I would have used the mean difference i.e. the mean difference was 300 ug L-1 but CLES was 0.55, I am unsure if the mean difference was very useful. So, from my naïve point of view, CLES provides relative reasonable information in a single number. Of course, we could us the mean and sd of both species or the mean difference and sd, but then if I am outside CLES gives in a single number a quite reasonable guess.

Wim Kaijser Thank you for your detailed explanation. Ecology is beyond my expertise. Indeed, if you have very noise data (field conditions), CLES may tend to be close to 0.5, even if the mean difference is "large". Unlike the p-value that decreases as the sample size increases, CLES (or EPa(0)) does not depend on the sample size. This is, I think, a good property of CLES.

Rainer Duesing I got curious to the EasyABC [1] package for R (perhaps interesting for you as well? it is also easy to use). If fumbled it in there and works relatively well (see attachment), albeit a bit slower than JAGS. However, it still bothers me that CLES only works when the data approximates a normal distribution. That is why I was so focused on the ranks. Although it seems that a simple log transformation works in most cases to get the data relatively normally distributed. So for relative straight forward comparisons it is not such an issue.

1. https://cran.r-project.org/web/packages/EasyABC/vignettes/EasyABC.pdf

Dear Wim Kaijser

many thanks for the code. I typically use the brms package for bayesian calculations, since it is very flexible and a lot of possibilities to handle the data are built in. I used a t-distribution model and a gaussian model and I came to very similar results as your approach. (typically CLES CI [.60, .90] roundabout). But I am a bit puzzled about the MCMC results, since there seems to be odd results, there are rows with identical values, which seems cumbersome to me. I do not know what the abc package does here.

Rainer Duesing Yes, I noticed the same, but forgot about it, still trying to figure it out (the repository is read only, would have been faster if it was not). I have seen people removing the first [5000] lines (somehow treating it as a burn in period) https://stats.stackexchange.com/questions/277499/simple-linear-regression-using-approximate-bayesian-computation-abc. However, this does not remove the duplicates. The trace plot looks otherwise okay.

Additionally, the ABC_rejection (and others) is in there as well which does not have this issue. However, its slower and results are slightly different (and a linear correction is needed). The lower interval is slightly higher than the previous. Another issue is that it gives a super weird posterior when you want to use multiple clusters (e.g. use_seed=T, n_cluster=6). It is just an interesting package.

Rainer Duesing Found the answer to: "there are rows with identical values, which seems cumbersome to me. I do not know what the abc package does here."

The duplicates are the cause of "sticking" to the previous prior and not "mixing" properly of the MCMC chain. It is best visible in Table Algorithm 2 MCMC-ABC inArticle Multilevel rejection sampling for approximate Bayesian computation

Where the distance (epsilon) of the simulate to the observed values is too large, than the previous accepted prior is added to the chain. Also, when the Hastings ratio is too small the previous prior is again added to the chain. Only when distance