Do we need a safety belt in cars? - Yes, "when properly applied".

For safety belts it's clear to most of us how they are properly applied. It's obvious that it would not be effective* when wrapped around the seat or around your wrist or your neck. Wrongly applied they can really do harm.

The situation (how to apply properly) is much less clear for significance tests.

That statistical significance is (almost) the only criterion required to make bold claims and to get work published is a misbehaviour of the scientific community (reviewers, editors, and, yes: readers who cite such papers). It puts significance tests in an inappropriate place, like putting a safety belt on the roof of a car so that everyone can see that this car has one (and hence is safe). It also does not really matter to show that your car has four safety belts if the task was to ship on a sea and it is ignored that a car that does have a properly applied safety belt actually drives on the correct road and in the right direction (i.e., using inappropriate research methods, experimental designs, or analysis strategies, or not testing the relevant hypotheses).

---

* well, at least not as intended

Tyler VanderWeelee has proposed E-value.

https://www.degruyter.com/document/doi/10.1515/jci-2018-0007/html

Regardless of the answers and what is right or wrong, thank you very much for sharing this text.

What I do know is that instead of placing cut-offs for probability values such as 0.05 or 0.01, we will be dealing with flexible and continuous probability values in the range of zero to one.

D. Eastern Kang Sim I dont see how E-value is directly related to the original question, which asks for the use of statistical significance and p-values. E-value is proposed to estimate the needed strength of association of a confounder variable, to explain away the association of your focal variable and the criterion. So it is not meant to replace p-values or anything. Please correct me, if I am wrong.

Hening Huang ,

I would support the conclusion in your quote that "P-values and significance tests, when properly applied and interpreted, increase the rigor of the conclusions drawn from data."

The reason being is that misapplication of statistics is abundant. Even if we do away with P-values people will find ways to get wrong Bayes Factors as well as any other alternative that is proposed. My take is that eliminating P-values will not solve the problem. On the contrary it will eliminate a useful tool that when properly used can be effective.

Regards

I traced some of the arguments to Goodman's criticism [1] of P-values. In the paper he states that:

"The root cause of our problem is a philosophy of scientific inference that is in dominant supported by the statistical methodology use. This philosophy might best be described as a form of "naive inductivism," a belief that all scientists seeing the same data should come to the same conclusions. By implication, anyone who draws a different conclusion must be doing so for nonscientific reasons. It takes as given the statistical models we impose on data, and treats the estimated parameters of such models as direct mirrors of reality rather than as highly filtered and potentially distorted views. It is a belief that scientific reasoning requires little more than statistical model fitting, or in our case, reporting odds ratios, P-values and the like, to arrive at the truth."

I find this interpretation in itself is also a bit simplistic on the assumptions that a scientist makes with respect to the tools of statistics. While some scientist may take this approach , I do not think it is a general view. I would tend to pin it more on misuse out of lack of knowledge rather than philosophical approach.

Just my opinion

[1]Goodman, S. N. (2001). Of P-values and Bayes: a modest proposal. Epidemiology, 12(3), 295-297.

There is no place in a scientific conclusion for statistical significance and p-values.

Statistical significance is a criterion one might chose in data analysis. As a criterion it is the lowest qualifying level for indicating a possible data observation. Achieving the qualifying level is not the prize; it is the entry fee for qualifying. The claim of a scientific prize at this level of data analysis is the grossest distortion of science. This obscene practice must be abandoned.

Yes. Scientists need statistical significance in data analysis to establish the base level of data for purposes of experimental design and development of assumptions.

I think one thing everyone (scientists and statisticians) agrees on is that statistical significance and p-values can be easily misinterpreted and misused. In order to publish papers, people tend to pursue p

Hening Huang It is not that p-values are easily misused or that reviewers have difficulty identifying misuse. The problem is two-fold.

• Cook book statistics teaches conclusion making from p-values
• Journals accept conclusions based on p-values

P-values are for data analysis and establishing the lowest value indicating a possible observation based on the data.

Joseph L Alvarez

I agree with you: "There is no place in a scientific conclusion for statistical significance and p-values." Indeed, scientists want to know whether their findings have "scientific significance" rather than "statistical significance". However, I am not sure what you mean by "P-values are for data analysis and establishing the lowest value indicating a possible observation based on the data". Do you mean that a p-value, e.g. p

Hening Huang

The p-value does not produce a meaningful scientific result, it produces a data quality result. The p-value is a data quality requirement set by the experimenter. The usual cook-book language starts with the null hypothesis and uses the p-value to reject the null. The accept-reject nonsense from cook books diverts thinking away from the experimental process.

Do the statistics first. Examine the the proposed experiment physically and statistically. Determine from the experiment and proposed statistical analysis the expected outcome. What minimum statistical value of the data will indicate the direction of the outcome. Set the minimum quality level that you, the experimenter, will accept as the p-value. Design the minimum experiment to meet this minimum data value. Perform an experiment better than the minimum.

Analyze the data at the conclusion of the experiment. Analyze the data, not the science. It is the data you compare to the p-value. Decide based on the data and the science how to proceed.

Note: This requires a little more thought than examples and homework found in statistical cook books.

Joseph L Alvarez Thank you very much for your further explanation.

Hening Huang

A p-value = alpha is exactly equivalent to what is considered a detection limit in metrology. The detection limit is the lowest value accepted by the experimenter that is discernible. That value may be counts, voltage, distance, etc., that is the output of the measuring device. The detection limit is based on device data, strictly. Device response is extended to a calibrated outcome, say, concentration, light intensity, size, etc. The basic data is lost in the calibration process. calibration detection limit is established. The calibrated data is used for a further process, say control, safety, cost limit, etc. Process limits are imposed and all decisions are made within the process that uses the calibrated data.

The process is remote from the measuring device and has not the least concept of the data. Good practice is to have the normal operational range of the measurement device at least 10 times higher than the detection limit. Nevertheless, process practice trends lower, toward the detection limit. The detection limit becomes the prize, not the level where control is lost.

This is the problem with considering the p-value

Joseph L Alvarez , I don't agree with this analogy p-value = detection limit. If you want to make this kind of analgy, I'd rather say that alpha = detection limit. The p-value is calculated from a sample. It's like a measured value, that can happen to be above or below a detection limit. I think this distinction is relevant.

I do not like p-values.

I prefer Confidence Interval, completed by the "distribution of the data"...

There is a discussion on the same matter in the Question

• Is “Statistical Significance” outdated?

This paper hereArticle Popularity leads to bad habits: Alternatives to “the statist...

There are many hypotheses for which it is hard to find an alternative to p-values, at least within the scope of classical (frequentist) statistics. For example, I do not see how one would compute a confidence interval for a goodness-of-fit test, or for a homogeneity test, or for stationarity tests in time series analysis

Dorothy Bishop published an interesting article about this issue actually, in Nature: https://www.nature.com/articles/d41586-020-02275-8

I was once told that "Science stops where Statistics start". Keeping in mind that statistics are also science, I can understand this point of view as we should not rely on these tools to draw any conclusion as it was stated above, but just to show the robustness of our data.

However, even in this case, we should also keep in mind that, so many statistics tools exist, that you can always find one that is showing what you want to show.

"Tell me what you want to show, and I'll give you the statistical test that you need."

This is why, as stated above and in Bishop's paper, you should not use any statistical test that you don't understand, just because it is common to use them, or because you are requested to. You have to be sure that the test you're using can be used on your data, and that you actually need it to show the robustness of your data. Maybe we should just let statisticians handle these tools after all? I understand that many reviewers and editors want robust data to be published, but not everyone is a statistics expert, and not every study needs it.

Up to now, the p-value has been used as a benchmark for statistical evaluations of data. If you have a statistically significant comparison, you can prefer it which you want. Moreover, in contrast to science, I am conservative in using the p-value. Especially, the p-value should be preferred for fixed effects.

Muff et al. proposed an alternative (published in Trends in Ecology and Evolution) to the binary language associated to p-values that we commonly used: https://www.cell.com/trends/ecology-evolution/fulltext/S0169-5347(21)00284-6

What's wrong with P-values is the fact that we use it as a sharp threshold to draw dichotomous conclusions, and that said threshold is completely arbitrary, and instead should probably be adapted considering each data set (depending on sample size, general noise in the data, or level of inter-individual variability...).

There are plenty of statistical tools that can be used in addition to or instead of p-values: but in the end, stats is not math, it uses math. So, no matter what stat test we use, WE give meaning to the statistical parameters that we derive from calculations. Math calculate the parameters but does not interpret them.

“A mini-literature-review: What have been said about Null Hypothesis Significance Test (NHST)“ Presentation A mini-literature-review: What have been said about Null Hyp...

A funny thing happened to a few scientists recently: They found philosophy and said ... maybe? I concur with the ASA's amended statement and Chloé Warret Rodrigues from her March 25th reply, that stats are not math. In philosophy of art and photography, this is equivalent to images of people NOT being people, but people agreeing that the experience of viewing clouds in photographs are equivalent to viewing clouds, if given enough to drink on those occasions and one manages to forego the fact that one might also be standing in a museum to view cloud images indoors. If we get so excited about equivalents, and continue down paths that do not have a care to real meat, we may find one day that we have ingested nothing for lunch, and instead, purchased a plate full of air in a noisy cafeteria and called it diet. But I digress: Science can't do that. Only people can do that.

I prefer using the coefficient of variation and it is mostly used by many journals. CV less than 20 is acceptable.