I have two distributions that I want to test for equality. The objective of traditional hypothesis testing is to disprove a null-hypothesis which states a lack of difference (H0: the distributions are not different). I want to test precisely the opposite: to prove the equality of two distributions . The problem: the null-hypothesis can never be proven, only rejected. So with traditional hypothesis testing (also called power approach) I could never claim that the distributions are actually equal but only that there is not enough evidence to claim they are different.
Equivalence Tests try to disprove a null-hypothesis which states a lack of equivalence (H0: the distributions are not equivalent). If the data provides enough evidence to reject the null-hypothesis, the alternative hypothesis can be assumed (H1: the distributions are equivalent). Equivalence tests are frequently used in pharmacology (to test whether a new treatment, which has less side effects than the old treatment, has also the same effect on the disease) or in sensory studies (does food A taste the same way as food B, meat substitute, etc).
I have never applied equivalence tests, which is why I am stuck now. I would like to use the Komogorov-Smirnov test to test the equality of two distributions. How could I do so using equivalence testing? I’m working in R.
Also: I would preferably do so without setting (arbitrary) equivalence margins. I found a method called Least equivalent allowable differences (LEAD), which, in theory, exactly tries to bypass setting arbitrary equivalence margins, but I have no idea how to implement this into a Komogorov-Smirnov Test. Has somebody ever done this before?
Maybe this interactive visualization might help to understand the rationale behind equivalence testing. ( https://rpsychologist.com/d3/equivalence/). And this post deals with a very similar problem, unfortunately without any comments (https://stats.stackexchange.com/questions/353767/kolmogorov-smirnov-tost-based-on-confidence-bands).
Article Understanding Equivalence and Noninferiority Testing
Article Least equivalent allowable differences in equivalence testing
The thing that confuses students is that here H0 and H1 are the reverse of what most students expect, i.e.
"The two sample Kolmogorov-Smirnov test is a nonparametric test that compares the cumulative distributions of two data sets(1,2). ... The null hypothesis is that both groups were sampled from populations with identical distributions."
This was taken from the link:
https://www.google.com/search?q=kolmogorov%E2%80%93smirnov+test&rlz=1C1CHBF_enUS847US847&oq=Kolmogorov&aqs=chrome.4.69i57j0l5.12502j0j8&sourceid=chrome&ie=UTF-8
and is true because d (the test statistic) is a measure of the distance between the two cdfs. Now in great, gory detail:
Thus if you choose alpha with corresponding p value, p0, then if p >p0 we reject H0 and conclude H1(the distributions are most likely different) . In the other case, say p
Please follow this link, I think you'll find your answer
https://stats.stackexchange.com/questions/97556/is-there-a-simple-equivalence-test-version-of-the-kolmogorov-smirnov-test
Safaa Kadhem thank you very much for this post. I like the idea of bootstrapping to build a confidence interval for the test-statistic θ:=supt|FX(t)−FY(t)| (the vertical difference of the two ECDFs). I would have two follow up questions:
- I'm still lost when it comes to specify the margins [−Δ, Δ]. I'm afraid to just select an arbitrary value. Is there a way to test for equivalence which bypasses margins? Least equivalent allowable differences (LEAD)? I mean the lower margin is actually easy: it should be 0 right? Because 0 would mean the two distributions are exactly the same. But I would need to define an upper bound.
- When bootstrapping, can I A) shuffle the data from all participants and then draw a subset of data consisting of datapoints from all participants, or B) do I do bootstrapping within each participant individually, leading to N confidence intervals and N statistical tests, where N is the number of participants?
Best,
Sven
- Badges
- Science topic