In order to examine whether significant difference exist in perceived service quality with respect to occupation I have applied Kruskal-Wallis (H Test), and got significant p value i.e. (H (2) = 5.972, p = 0.050). Since I have three categories of occupation I applied post hoc, but before that I calculated effect size estimate. To calculate Effect size estimate, I divided chi-square value = 5.972 with N-1 (I have 308 respondents so N-1=307). So the value of effect size estimate is = 1.9%). Is it a decent effect size to report? Subsequent, Post hoc tests are significant.
I'd say (provacatively): if you can't interpret the effect size then how can you interpret your results? How did you plan your research/experiments if you don't have any idea what possible results would tell you?
You must break down what the effects you estimated will mean practically. This can not be expressed in terms of "significance" or other estimates (these would all be circular statements).
@Stephen: don't mix or confuse absolute and relative effect sizes. You mean that relative effect sizes can be small even when the absolute effect sizes are large but the noise is large, too. I suggest to avoid interpreting relative effect sizes at all, because this is much more difficult and potentially misleading, because two independent and individually important pieces of information (effect and noise) are intermingled. That's like interpeting p-values: it generally goes wrong...
I don't know the details of this statistic, but in general, whether an effect is "big" or not varies from field to field, and research question to research question. What's a big effect size is for one phenomenon might be a small effect size for another. For example, in social psychology, many things are measured with a lot of noise and therefore effect sizes tend to be pretty small; but even then, some very robust phenomena (like anchoring) have larger effect sizes. So there is no one hard-and-fast answer to what effect size is "big enough"; this depends on the details of your research question, and looking through the known literature to make common-sense estimate of what sort of effect size should be expected.
More importantly, the choice of whether or not to report an effect size should not depend on its magnitude. If the effect is of relevance to your research question, it should be reported, no matter how large or small it is, no matter whether it's consistent or inconsistent with your preferred hypothesis.
I'd say (provacatively): if you can't interpret the effect size then how can you interpret your results? How did you plan your research/experiments if you don't have any idea what possible results would tell you?
You must break down what the effects you estimated will mean practically. This can not be expressed in terms of "significance" or other estimates (these would all be circular statements).
@Stephen: don't mix or confuse absolute and relative effect sizes. You mean that relative effect sizes can be small even when the absolute effect sizes are large but the noise is large, too. I suggest to avoid interpreting relative effect sizes at all, because this is much more difficult and potentially misleading, because two independent and individually important pieces of information (effect and noise) are intermingled. That's like interpeting p-values: it generally goes wrong...
firstly your p-value is on boundary line, to get a more dependable result increase your sample size so that the p value can move to either side of the critical bound
I'm not sure that's a good idea. The p-value should be based on a pre-determined sample size, or at least a convenience sample size. But repeatedly adding participants until the p-value gets significant or not essentially invalidates its meaning anyway. See http://www.ncbi.nlm.nih.gov/pubmed/22006061.
There are ways to make something like this work (e.g. http://daniellakens.blogspot.co.uk/2014/06/data-peeking-without-p-hacking.html) but they also depend on making decisions beforehand, not on adjusting your sample post-hoc based on how the stats come out.
Here’s something funny. I was googling around for ways to calculate effect size statistics for Kruskal-Wallis, and I quickly found a couple of sources that had a calculation for epsilon-squared as = H / {(n2 – 1) / (n + 1)}, where n2 is n squared. So I started wondering to myself how far off this would be from the OP’s statistic of H / (n-1)…
There are many definitions of effect size. As Jochen said, you must be able to interpret your results and the needed magnitude of the results should be known before measurements begin.
The expression of effect size, whatever the definition, is based on the model of the data used. The uncertainty of the model is not the uncertainty of the data, but might be a close approximation. Nonparametric tests have further interpretations (or lack of interpretation) of uncertainty. Tests may obscure the structure of the data so the size of the test may have little actual meaning to the research purpose. You must decide how the test translates the outcome of the experiment. What does chi2/(N-1) mean?
Thank you Stephen, Jochen, Amechi and Joseph for your inputs. I referred to this video for calculations https://www.youtube.com/watch?v=xLAUs4hL5Vs
The formula you used is for Epsilon-squared, which is probably fine as a measure of effect size for the K-W test.
I wasn’t questioning the formula. I’ll explain my comment in another post.
But you may not want to cite a youtube video as reference!
This paper is a useful source for computing effect sizes:
http://www.tss.awf.poznan.pl/files/3_Trends_Vol21_2014__no1_20.pdf
Also, the following link is from Statistical Reasoning in the Behavioral Sciences. by Bruce M. King. 6th Edition. It is a good source for effect sizes for non-parametric statistics. I can’t attest that the link is authorized by the author or publisher.
http://ebooks.narotama.ac.id/files/Statistical%20Reasoning%20in%20the%20Behavioral%20Sciences%20(6th%20Edition)/Chapter%2021%20%20Some%20(Almost)%20Assumption-Free%20Tests.pdf
- Badges
- Science topic