Recently there seems to be an increasing discussion about the use (and misuse) of the p-value and statistical significance. This discussion is not new, of course.
One of the most widespread recommendations is to present not only the p-value obtained, but also the confidence intervals and measures of effect size. While this is a recommendation that improves the presentation of results, researchers generally use arbitrary and unjustified benchmarks for these effect size measures, citing Cohen's traditional work.
In this sense one author who has worked a lot on this subject is Daniel Lakens. In its article "Calculating and reporting effect sizes to facilitate cumulative science…" talking about Cohen's d, indicates that:
"these values are arbitrary and should not be interpreted rigidly... The only reason to use these benchmarks is because findings are extremely novel, and cannot be compared to related findings in the literature..."
In this line, Funder and Ozer in their paper "Evaluating Effect Size in Psychological Research: Sense and Nonsense", propose a more justified way of interpreting the effect size.
Finally, agreement with the group of Daniel Lakens, we should move from an obsetion for finding rigid and arbitrary cut-off points to "...transparently report and justify all choices (...) when designing a study...".
Hi Pablo,
I agree with confidence intervals and effect sizes as a better way of presenting results. I obviously see problems when we interpret results as significant/insignificant just because they exceed a certain value
But even if we don't misuse these methods (P values or Confidence Intervals) there is still a substantial problem.
Applied statistics nowadays is mainly frequentist in the Neyman-Pearson (NP) sense, i.e. methods that have low error probabilities in the long run.
(you can use a statistic, i.e. confidence intervals or pvalues, to accept the null or the alternative hypothesis).
These methods do not get along with the Likelihood principle very well.
This leads to one basic problem: confidence intervals (typically 95%) are pre-data intervals, hence they tell you nothing about the parameters you are trying to make inference on when you actually look at the data.
That is why, when you have an interval after you observe the data, you know that the probability that the parameter is covered by the interval is either 1 or 0. Which is not what we are after, unless we are in quality control.
i.e. we would like to have an interval that covers the parameter with a given probability, but this can only be done using fiducial intervals or credibility intervals, that can be interpreted epistemically
Confidence intervals cannot be interpreted evidentially.
That is why the NP theory is often referred to as "inductive behavior".
https://link.springer.com/content/pdf/10.1007%2FBF00485352.pdf
http://bjps.oxfordjournals.org/content/early/2014/03/26/bjps.axt039.short?rss=1
You have to ask a question about an effect. Estimate how large of an effect will be important. Estimate the uncertainty in determining the effect. Estimate the sample size to measure the effect in the presence of the uncertainty (signal above the noise.)
The analysis of the completed experiment requires the usual statistical measures to ensure data quality and data quality minimums.
The resulting signal above the noise, effect size, must be large enough to quantify. Quantify means an effect size much larger than the noise. The noise includes all the uncertainties in the experimental conditions and assumptions; fully propagated uncertainty.
The effect size must be large enough that a prediction of future events can be made with control. Control means a predicted outcome as 9 out of 10 on a defined population.
Thanks Stefano Nembrini and Joseph L Alvarez for your comments and reference articles.
I recommended the answer provided by Stefano Nembrini
Best Regards Pablo Ezequiel Flores Kanter
Effect sizes are obviously linked to pvalues in a very straightforward way so beware. p values functions and confidence distributions as well as bootstrap distributions and posterior distributions with different priors may be more informative. Best
- Badges
- Science topic
- Similar topics
- Geoscience
- Asia
- India