Trying to remember a quote by John Tukey, I found it on AZQuotes:

"An approximate answer to the right problem is worth a good deal more than an exact answer to an approximate problem." 

How might this fit to a degree into the overall question here? 

Dear James,

Thank you for your interesting question. There are many aspects to it so I am not going to attempt to answer it fully. Statistical testing is a means to making a decision. It leads to saying something definitive about a data set. Obviously such a decision may turn out to be wrong (with the benefit of additional data) and should be interpreted cautiously. We encourage researchers to talk informally about the strength of evidence as well as the alpha level statistical decision.

Another aspect you correctly mention is the appropriateness of a particular test. Again, the decision of whether to use a parametric test needs to be made but it is not black and white. Robustness exceptions can indicate when a parametric test is still valid even when its assumptions are violated. Sometimes we recommend using both a parametric test and a nonparametric one.

I agree that there can be an overemphasis on null hypothesis statistical testing but from a pedagogical perspective it is often the next thing to teach after descriptive statistics.

I have not yet seen any instance where a significance test added any useful value beyond a careful examination and interpretation of the data (in the context of the experimental design and the research).

It is - sadly! - a hard requirement to get manuscripts published. Following non-sensical and even illogical standard procedures seems to be "more scientific" than a careful examination and interpretation of the data.

I have not seen any example where hypothesis tests were perfomred and where any comprehensible argument for the choice of alpha and beta was given. It is therefore unclear if the decision strategy is rational in some sense. It is common in animal experiments that the animal numbers required for a study must be planned according to a hypothesis test. The crucial point is that researchers can not give any cost function of a decision, they often can not even identify alternatives that would be worth to be distinguished. So the usual habit is to tune the parameters (mostly the effect size) to get som a priori desired number of animals. This is clearly undermining the logic of the whole procedure.

In screening experiments I use p-values to establish a ranking of candidates, but I would not use them to make a yes/no decision. This way I only sort the candidates by their "statistical signal-to-noise ratio" (what p measures: lower p-values indicate a higher stat. SNR), but I don't use any arbitrary cut-off for a yes/no decision, and there is no need to control error-rates. A selection of candidates is oriented at the number of candidates that can be studied in further following experiments, where much more data per candidate can be generated and be examined and interpreted.

PS:

@Peter: I think James did not mean to discuss the "appropriateness of a particular test" but rather the appropriateness of a testing procedure at all.

"....the 'appropriateness of a particular test' [or] ... the appropriateness of a testing procedure at all."  -  I'd say the former is often one of the various problems with the latter. 

I agree with James and that is why I recommended his views.

Thank you for that question.

One aspect, that I get from it, is the critique of the inflated use of and reliance on significance tests to validate empirical results. I agree with that.

Nevertheless, I think a statistical evaluation is mandatory, if we want to avoid starting to interpret noise. 

The purpose of a statistical test (null hypothesis significance test) is to evaluate whether we have enough evidence to conclude that an effect, say some positive correlation, we observe in a sample, is sufficiently unlikely to be observed if the correlation in the population is actually negative. If we don't have enough evidence to support that statement, it does not make much sense to draw any firm conclusions.

However, having a significant effect is only the basic reqirement. It is a bit like when you go to the doctor and, after hours of diagnostics he comes up with the result: you are not dead. This may be true but not very suprising. However, if you were not alive, further treatment would be unnecessary.

Similar with scientific results. The question if it is significant or not is usually not relevant. Relevant is how the result may support well founded (subject matter) hypothesis and is relevant to advance the formation of some theory. But this may only be based on results of which we can at least state that the observation of a positive effect in a sample very likely reflects a positive effect in the population under study

The problem with "significance" (a misnomer) is that without paying attention to a type II error probability analysis, it is routinely misused.  A confidence interval can often be much more usefully interpretable. 

Consider https://www.researchgate.net/publication/262971440_Practical_Interpretation_of_Hypothesis_Tests_-_letter_to_the_editor_-_TAS

Article Practical Interpretation of Hypothesis Tests - letter to the...

Trying to remember a quote by John Tukey, I found it on AZQuotes:

"An approximate answer to the right problem is worth a good deal more than an exact answer to an approximate problem." 

How might this fit to a degree into the overall question here? 

James,

I agree with you in principle (especially misinterpretation of statistical tests is surely a problem). But I think for the inference problem of evaluating if observed evidence supports some conclusion about a population - e.g. a sample showing positive correlation between some variables as evidence for a positive correlation in the respective population - NHST and confidence intervals are equivalent.

That means, if for the above inference problem you would use a confidence interval (eg by interpreting the lower limit of the interval being greater then 0 as evidence for a positive correlation), this would still be a test to some type I level alpha.

Or did I mißunderstand you point?

Whenever you could use a confidence interval, the same information might be presented, but perhaps more awkwardly, by a combination of type I and type II error probabilities. 

The point here, however, is that statistics may too often - in my opinion - be thought of as a set of tools for providing yes or no answers.  We have to remember that we are dealing with random variables.  Models, notably/famously, are never exactly true.  Perhaps we think too much in terms of absolutes, leading, for example, for one to assume a model is homoscedastic or not ... yes or no ... rather than just estimating the degree of heteroscedasticity (based on the data), and robustness considerations.  Often a question on ResearchGate will say that this and that have been 'tested for,' as if everything were a matter of yes/no, rather than degree.  Even in something more deterministic, such as operations research problems, there are bound to be complicated situations that might require something other than yes/no thinking.

This is just what I seem to have observed, and I think it goes beyond consideration of models and/or hypotheses. 

...

(For one thing I'm suggesting that estimation and attention to standard deviations and standard errors is more important than p-values, though a p-value uses them.)

For frequentist statistics, almost all statistical significance tests are affected so much by sample size. If your sample size is big enough, "all" statistical tests generates significant p values, even there is no difference existing. This situation is called by several fun names like; data fishing, p hacking, data dredging and data snooping. While you have huge number of samples, the sample size becomes the only parameter for defining significance instead of main descriptive statistics parameters like mean, mode, median or deviation. So, statistical significance tests (meaning p values) becomes useless for big data sets.

An experimenter proposes a hyothesis based on obseration and established theory and knowledge. This prior information is used to assert a conditional outcome. The experimenter uses the hypothesis to state given (prior information) and to prove (some outcome conditioned on the prior information). It is incombent onpon the experimeter to qualifiy and quantify what is proof. Statistical definitions, requirments, and tests provide bases for quantifying what constitutes proof. Nevertheless, proof is not attainment of some statistical criterion.

The 'To Prove' statement contains the criteria for proof. The necessary values or differences needed to demonstrate proof are established before data analysis. The type of analysis is part of the criteria for proof. The statistical tests or procedures used in the analysis are based on the the expected data and the values or differences needed to demonstrate proof.

Null hypothesis significance testing based on a p-value or confidence intervals is a minimum criterion and much too uncertain to draw a conclusion. The specified test or procedure does not usually include the fully propagated uncertainty within the procedure, instead, the uncertainty is limited to the distribution of the procedure. Nevertheless, the 95th percentile of the procedure distribution renders the decision highly uncertain. Include the total propagated uncertainty and the decision is farsical.

Achieving a 95th percentile in a statistical procedure might be the basis for continued investigation, but it cannot be the basis for concluding success. The criteria for proof are the bases for concluding success and are what is used to defend the conclusion.

p-values do not stand alone.  They do not mean what people often think they mean.  Setting "significance" everywhere at 0.05 is a big mistake. (If "Big Data" were feasible long ago, the equally bad mistake of ubiquitous use of something like 0.0001 might have become entrenched.) 

Note the letter to The American Statistician which I attached earlier.

Does anyone have comments outside of the use of hypothesis tests (or much worse, "significance")? 

Thank you.

Your question asked, "So, do you have an example of a project gone wrong because emphasis was placed on results from a 'statistical test'?"

Your last comment asked, "Does anyone have comments outside of the use of hypothesis tests (or much worse, "significance")?"

How else do you score a statistical test?

If you still mean,"So, do you have an example of a project gone wrong because emphasis was placed on results from a 'statistical test'?"

I could list many examples, particularly in the field of environmental contamination. Possibly, the most egregious example comes from the medical field, the conclusion that cholesterol lowering drugs reduce heart attacks. A full-page ad in the Wall Street Journal announced, "Lipitor reduces risk of heart attack by 36%*". The asterisked footnote stated,"*That means in a large clinical study, 3% of patients taking a sugar pill or placebo had a heart attack compared to 2% of patients taking Lipitor." 

This claim of 3% vs 2% passed a statistical test. This claim means that 100 people must be treated to prevent 1 heart attack. Not a very good result. It was also reported that 3% of participants stopped taking the statin because of side effects. So 100 people must be treated to produce 3 side effects. Not bad if the 100 were prevented heart attacks.

The add was placed in 2007, last year a review article was published,

thelancet.com Published online September 8, 2016 http://dx.doi.org/10.1016/S0140-6736(16)31357-5,

that purported to show that statin use by 10 patients prevented 1 heart attack after a prior heart attack or 20 at risk patients must be treated with no prior heart attack to prevent 1. Side effects were much lower than the earlier claim. While the results sound better, the review performed many contortions to come up with a less than favorable number needed to treat.

Disclaimer: I have studied the heart/cholesterol claims on my own since the late 1960s. I have total cholesterol >350 mg/dL. I have found no medically significant reason to lower my cholesterol. I received no financial support nor do I have interest in any medical or pharmaceutical enterprise or research.

Null hypothesis significance testing (NHST) has been under fire for more than half a century (and Fisher vs. Neyman & Pearson predates that).  What are the virtues of NHST?  No other decision-making system has been contrived to answer a specific (though typically not very informative) question with known (and controllable) risks of being incorrect in the decision.  That's about it, I think.  What are the drawbacks?  (a) The null hypothesis is always wrong; (b) Outcome depends on sample size, and therefore both massive differences can go undetected (with too-small N) and trivial differences can be detected (with large N); (c) The desired question isn't really answered; and (d) There is frequently a disconnect between the judgment of statistical significance (non-chance result, under Ho) and that of practical/clinical/policy-making significance.  If gasoline prices change by 1 penny per unit (liter, gallon), then the result will be noteworthy for the vendor who buys/sells 1M or more units per week, but negligible for the ordinary driver filling up his/her vehicle.

Are there better ways?  (a) Parameter estimates via confidence intervals; (b) Effect size estimates (which are independent of N); and (c) Bayesian methods (entertaining the likelihood of multiple options as outcomes), to name some obvious alternatives.  The text, What if there were no significance tests? (Harlow, Mulaik, & Steiger, Eds.) lays all these out quite nicely.

However, the method chosen doesn't protect against people who would, in the words of Darrell Huff, seek to "lie with statistics," as with Joseph Alvarez's example, above.  Or, from promulgating accidental results as fact, such as Tyler Vigen's amusing examples of spurious correlations: http://www.tylervigen.com/spurious-correlations  

Cute graphics, but not unexpected since they aren't graphing variables against each other, but that various variables might each tend to increase (or each tend to decrease) with time.  :-)

@David,

nice post. I have to comment on this sentence (not relevant for you but more for other readers):

"No other decision-making system has been contrived to answer a specific (though typically not very informative) question with known (and controllable) risks of being incorrect in the decision."

Noteably, these controlled error rates refer to a model with a fixed hypothesis. If the null model is correct then the rate of observing p

What about "information criteria," such as the AIC?   (I am thinking of the term 'statistical test' in a very broad sense.  -  See https://www.researchgate.net/post/How_can_I_compare_predictive_power_of_two_regression_models.) 

What about graphical 'tests,' such as residual analyses? 

.....

Examples of using graphics involving estimated residuals:

 https://onlinecourses.science.psu.edu/stat501/node/2

 https://www.researchgate.net/publication/313925533_Comparing_Model_Performances_Graphically

 https://www.researchgate.net/publication/263809034_Alternative_to_the_Iterated_Reweighted_Least_Squares_Method_-_Apparent_Heteroscedasticity_and_Linear_Regression_Model_Sampling

Method Comparing Model Performances Graphically

Conference Paper Alternative to the Iterated Reweighted Least Squares Method ...

Disadvantages of treating the "definition" of R-square literally in practice?  (I mean, impacts that make it's estimate misleading.)  

From the above three notes, you can see that I mean to ask about more than "significance."   

Your comments?  

Thank you.  

I am astounded that so many people see a one in twenty chance (5%) of being wrong as absolute truth. As a young graduate student I actually saw two of my professors call each other 'liar' because they got different results on  tests involving 5% statistical significance. Even though I have written a manuscript for a Neyman-Pearson based undergraduate texts which is intended to cut the cost of a basic text by 2/3, I agree with a lot of the criticism of the approach as practiced. However, due to the lack of attention to Type II error the vast majority of sample sizes in most research are too small.  A few years back more than one author reported that about 80% of orthopedic research (we are talking experimental human subject surgery here!) used inadequate sample sizes.

I try to show beginning students how to combine validity with the, not always reasonable demands of the journals. So, I only speculate on the fact that, confidence intervals are almost certainly superior to criteria based decisions. However, adapting the Neyman-Pearson approach to estimate sample size as though you intended to make a criteria based decision will tend to give you considerable control over your confidence interval. That is it will rarely come out too wide to be of use if you consider the Type II error first. However, that leaves you with a need for an estimate of variability requiring referral to records or a need for a pilot study to estimate the variability.  

Many (most?) researchers treat the statistics as magic and are unwilling to put so much effort into it. The question might really be, is the problem the statistical approach we use, or the researchers knowledge of it, and attitude towards it. 

PS: Beware of "small", "medium", and "large" indexes, they are misleading and even their author has admitted it.  

Wow lots of detailed comments and bashing of the p-value!

Has anyone mentioned that for many/most tests the p-value and confidence interval are analogous?  And therefore instead of thinking in 'significance' with p-values one can simply think in terms of confidence intervals and the uncertainty quantification they represent? Large p-values mean that a confidence interval will capture 0, and therefore you cannot reliable measure the parameter of interest. Sort of inline with @Jochen Wilhelm's first paragraph where he states 'p-value adds nothing compared to careful inspection of the data'. Of course!

Stephen - 

You say "... for many/most tests the p-value and confidence interval are analogous" - Well, you forget type II error probabilities, which is the point.

For that reason, a sequential hypothesis test between two specific hypotheses is somewhat more analogous to a confidence interval.

Jim

PS - You can do OK with a p-value and a type II error analysis, but interpretability is often problematic.