Assuming we have six vectors: A, B, C, A', B', and C'. A, B, and C are the outputs of method X. A', B', and C' are the outputs of method Y. These methods are iterative, and the outputs are gathered at certain iterations for several experiment repetitions. For example, let's assume A = [a1, a2, ..., an], containing the outputs of running X at the 10th iteration in multiple experiments. Then, a1 is the output of the first experiment, a2 is the output of the second one, and so on.
We tend to check whether X and Y have different means at the specific iterations using the Wilcoxon Rank Sum test. Therefore, the comparisons are as follow:
A vs A' (e.g., the outputs of 10th iteration)
B vs B' (e.g., the outputs of 20th iteration)
C vs C' (e.g., the outputs of 30th iteration)
The question is that although we are performing multiple comparisons, the vectors are separate samples and different from each other. Thus, do we still need significance level correction? If yes, which correction method would you suggest?
See the attached for when you use and when you don't. These are some of the most used tests in the natural sciences. Further they're fairly new. Best wishes, David Booth
See these links:
https://www.researchgate.net/post/Should-we-use-a-Boneferroni-correction-to-adjust-the-sgnificance-level-when-performing-Wilcoxon-statistic-for-multiple-pair-comparisons
https://sphweb.bumc.bu.edu/otlt/mph-modules/bs/bs704_nonparametric/bs704_nonparametric4.html
https://www.sciencedirect.com/topics/medicine-and-dentistry/wilcoxon-signed-ranks-test
https://www.sciencedirect.com/topics/mathematics/wilcoxon-signed-rank-test
https://nyuwinthrop.org/wp-content/uploads/2019/08/wilcoxon-rank-sum-test.pdf
This only depends on if you want to control the FWER or not.
Apart from this I think the question is not well posed. Imho it would be more sense to analyze the relationship between the difference depending on the number of iterations ("regression model").
- Badges
- Science topic
- Similar topics
- Mathematics
- Statistics