p - hacking and falsification of statistical results Improvement processes regarding process evaluations Foreword The significance of a statistical statement is denoted by p, as a probabilistic variable. The term "significance" translated into the Englsh version means "clarity, the essential". There is no question that significance as a measurement variable in probabilistic statistics plays an extraordinary role. Nevertheless, it is often subjected to manipulation by keeping the number of random variables - i.e. measured values - small, or even filtering them. In addition, the inadequate integration of all process parameters and the inadequate use of probability densities mean that processes are inadequately evaluated both now and in the future.
so what to do? suppressing data, those ones that are unliked? see pic p-hacking3
A plausible example, Fig.1.1, of this was given in Spectrum of Science SPECIAL 3.7, Chapter "Estimating Error, the Curse of the P-Value", Regina Nuzzo, Gallaudet University Washington.
Better way: include all data any make a frequency-scale and gain parameter values for a probabilitydensity see pic p-hacking2
That is ok?
My preferred approach to address p-hacking is for the experimenter to perform inference using raw p-values and confidence intervals, and to make tentative decisions using per-comparison error rates while acknowleding the total number of tests performed, noting that no hypothesis is proven false with a single small p-value, nor is it proven true with a large one. All we can do is provide the weight of the evidence.
Strong control of a family-wise error rate using multiplicity adjustments is certainly another way to go, but definitely not my favorite.
I also prefer to reserve the word falsification to refer to Popper's notion of falsifiability in science, rather than as a synonym for "fabrication."
I don't think there is a solution to p-hacking as long as p-values are still used. I think the only solution is to abandon the whole statistical significance testing paradigm, as suggested by many scientists and statisticians.
In the end, beyond the technical requirements for a specific statistical procedure (aka "pre-requisites"), it is the researcher's duty and responsibility to wisely select data, according to his/her knowledge and experience.
i.e. Removing outliers beyond a certain threshold based on own/field's experience does not mean tampering the data, while the results definitely are changed.
Reporting crude or adjusted values is, as well, the result of an educated choice, rather than a "mechanical" procedure: Usually, when direct or latent influences are suspected between the indicators, adjusted values are mandatory, even if, usually, we lose "significance".
There is no recipe, then, and any choice you make is reflected in the results. So be ready to have clear and sound explanations for those and, ofc, keep in mind that any result should be reproducible.
... perhaps we could use probabilitydensities that respect skewness, longer tails?
- Badges
- Science topic
- Similar topics
- Correction