I often work with large scale datasets (1000+ samples) obtained by monitoring's programs. I have expectations about patterns (based on literature and own experience) in this dataset which I analysis with various models. Critique has arisen regarding Null-Hypothesis-Significance-Testing (NHST). As such, the use of p-values is perfectly fine, it is, however, the interpretation and use of p < .05 that bothers me. I preferably use effect-size estimators with corresponding "confidence" intervals and p-values. I have the idea that some reviewers are perhaps less familiar with the concept of p-values.
As such, in the field I currently work p-values are still misinterpreted (neither am I perfect in this regard). I therefore refrain from the term "significant" or add a value laden term to the p-value. As such I describe the patterns and effect-sizes and and after this simply add (LCI = ...; HCI = ...; p < ....). The effect-size is my guiding image of which I interpret the magnitude. The expectation I have about this magnitude are thus "important". However, many articles in the field I work do not describe effect-sizes and my interpretation can only arise from my own work. Hence, small effect-sizes are less interesting, and it is perfectly possible to find a small effect-size of which p
I have to to admit that I don't understand that either. Best of luck in the future, D. Booth
I'm with you on this....
One way I sometimes think about statistical analysis is that there are three "statistics" to think about: 1) The p-value, "statistical significance"; 2) The effect size; 3) The "practical importance".
Each of these assesses a different thing, so each has meaning.
The p-value is like a "signal-versus-noise" measure. That's useful. Do we think that (probably) there's a real effect there?
The effect size is the size of the effect, however we might be measuring it. It doesn't have to be an "effect size statistic". Sometimes a simple difference in means or difference in medians is meaningful.
The practical importance is, well, the assessment of what is important in reality.
In one scenario, sometimes small effect sizes are meaningful, for example in cases of life or death, or in the example of a website that has many visitors and where a small percentage increase in sales is a lot of money. Even in politics, sometimes differences we would normally assess as small are treated like a big deal. (I have a real example here, search for "Pennsylvania", https://rcompanion.org/handbook/H_10.html ).
On the other hand, if we have a small sample, a large-ish p-value, but a large effect size, that's really suggestive. We often see this in preliminary studies with many treatments and small sample sizes. There's something there that maybe should be further investigated.
Finally, practical importance includes all kinds of other considerations: Costs; Externalities like environmental effects; the risk of life when new treatments are uncertain. In my career I can think of a couple of cases where statistically significant results that ignored practical importance that maybe would lead to a different conclusion.
For my general argument, I guess I have to cite myself here, if only for copyright reasons, though what I have on my website could use some revisions. C'est la vie. https://rcompanion.org/handbook/D_01.html .
p-value is a function of the signal-to-noise ratio that depends on the sample size. It does not measure the real magnitude of "effect size". Besides, some effect size such as Cohen’s d does not measure the absolute difference between two means. I suggest you try to use "relative mean effect size" (RMES) that is more meaningful than Cohen’s d. Discussion on RMES can be find in Preprint Exceedance probability analysis: a practical and effective a...
Hening Huang :
I would support reporting standardized effect sizes (with confidence intervals). However, one should note that there is some confusion about the terms, for example Cohen's d and Hedges' g are defined differently depending on the literature or software used. Additionally, it is not well understood that there are biased and unbiased variants, but one should report which version actually has been used. See Technical Report Notes on Effect Size Measures for the Difference of Means Fr...
Dirk Enzmann:
Thank you for sharing your technical report that helps clarify some confusion about the terms of standardized effect size. My problem with Cohen's d is that it seems lacking of physical meaning. To simplify the discussion, consider the case where n1=n2=n. The pooled standard deviation then becomes the square root of the mean of Var(X1) and Var(X2) (estimated). This pooled standard deviation is not the standard deviation of the difference between the two sample means, or it is not the noise in the estimated absolute effect size (the difference between the two sample means, i.e. the signal). Thus, Cohen's d is not a correct measure of the signal to noise ratio.
In my opinion, a meaningful effect size must have a clear physical meaning. There are three alternatives: (1) absolute effect size (magnitude, not standardized), (2) relative effect size such as RMES (relative magnitude with respect to a (consensus) mean), and (3) signal to noise ratio (standardized). RMES is based on a very common concept of the relative difference between two quantities of same nature. This concept is easy to understand and is widely used in practice. For example, a company plans to have an annual revenue growth by 15%. I am a little surprised that this concept is not often used in statistical inference.
- Badges
- Science method
- Similar topics
- Teaching