Confidence intervals,Bayesian methods, effect sizes are  alternatives for p values. There are other alternatives too.

Lots of resources on internet like the ones I attached.

http://theoryandscience.icaap.org/content/vol4.1/02_denis.html

https://www.msu.edu/~levinet/NHST2.pdf

http://psycnet.apa.org/books/12316/005

"P-values are not doing the job they were supposed to do."

That's not really correct. They do what they are supposed to do.

The problem is that

a) they are used and interpreted in ways they were not supposed to be used and interpreted* and

b) they are taken out of their context (what means: misuse or abuse) by un-linking the experimental design from the reporting and interpretation of p-values**

Both problems are related to the (good?) scientific practice. It is not the problem of the p-values as such. My concerns about "alternatives" is that they won't solve the fundamental problem(s): a lack of statistical thinking and not-so-good scientific practices.

---

* P-values are not indicating "reproducibility" or "evidence". That's already a misconception. P-values don't proof or disproof a hypothesis, that's another misconception. And P-values are not to "test competitive hypotheses". This misconceptions is related to the unfortunate promotion of the "null hypothesis significance test" (NHST) that is explained in so many (bad) statistics books - a procedure that is logically flawed. We should not be surprised to get useless results if we employ illogical procedures.

** like for instance: repeating an experiment until the p-value is low and reporting only this result; screening experimental conditions or correlations and reporting only those that showed small p-values; using different analyses and reporting only the one that showed a low p-value; reshaping the data, removing outliers etc. and reporting only the result with a low p-value; ....

@Jochen: Thank you for clarifying the issue. My upbringing is with those flawed statistics books. I don't know how we can think logically if our minds are clouded and preoccupied with logically flawed procedures. You just made me realize that my question was not even correct.

I cannot add anything more to the Jochen`s answer.

For your information, the added paper is a good explanation of the issue.

ciao

:)

You may now that the "P-value" was invented by R.A.Fisher as a tool for research workers to measure a "surprise index", nothing else but some conveniently normalized way to express a "signal to noise ratio" of an estimate (like a mean, or a difference, etc). The "signal" is taken with reference to an expectation under a "null"-hypothesis, and a large signal-to-noise ratio (-> a low p-value) is just used to justify that there is enough "signal" to be interpreted and that further experiments in this direction can expected to be fruitful. There is nothing like the control of an "error rate" - this concept of "falsely rejecting the null" is non-sensical. The question that is to be answered is just if the signal of the estimate is sufficiently large to practically justify the rejection of the null-hypothesis. It is NOT about the question whether or not the null hypothesis is false. It is surely false - this is known a priori, and testing this makes no sense.

This p-value is a rational number between 0 and 1. Its value is called the "significance" (of the observed data under a specified statistical model and a given null hypothesis). This is actually a continuous measure from "extremely significant" (p close to 0) to "not at all significant" (p-value close to 1). There is no magic cut-off that would distinguish a "significant" from a "non-significant" result. There are onle more or less significant data. However (and this is one source of confusion), if a reasercher decides to "reject the null hypothesis" (i.e. claiming that the observed signal is worth to be taken seriousely) this is often called a "significant result" and opposed to a "non-significant result" (i.e. the signal is not considered clear enough to be taken seriousely).  This is still not about controlling error-rates. It is still a -neccesarily subjective! - decision of the researcher, only expressing that he/she believes that there is enough relevant information in the observed data to take it seriously as a "signal". Surely, there is little point taking signals serious when the p-value is close to 1. The closer to zero, the better. But -depending on the entire context of the research and the experiment - sometimes a p-value of 0.2 may be enough to take a signal serious, and sometimes a p-values of 0.001 may not be sufficient at all. There is no general answer when to reject the null hypothesis, and the answer is not given onyl by the p-value. It is absolutely required to put this into a proper context.

As Fisher noted is the precise value not very intersting. In the field of genetic and agricultural research at the time of Fisher, a "rather low value", like somewhere around 0.05, might be a benchmark that could be reasonable in most cases. This should just not be blindly extended to other contexts (like, for example, clinical studies with 10000s of patients, or a screening of small-molecules in bioassays, or observational studies that are actually screenings for correlations etc.).

The value of this significance-measure is relatively limited. The good thing about it is that this is a normalized measure that can be interpreted statistically as the expectation to more extreme test-statistics than the one obtained from the actual sample, under the null-hypothesis (therefore I call it a "surprise index"). The bad things are:

* it does not add any information beyond the signal-to-noise ratio; one could interpret the signal-to-noise ratio more directly.

* deciding against the rejection of the null-hypothesis is nothing but the confession that the results are inconclusive

* the decision is done quite informal, relevant aspects that lead to the decision are often not mentioned or stated explicitely; this also impedes an assessment of the confidence of a decision.

J Neyman (and ES Pearson) did not agree with Fishers logic to use p-values for decisions. They developed a purely decision.theoretic approach that allowd to decide between two alternative hypotheses (A vs. B) with a pre-defined confidence. The aim is to make a rational decision betwene A and B. It is not about rejecting a null-hypothesis. Either one accepts A or one accepts B. The rationality of the decision is introduced by the attempt to maximize the expected utility of the decision. This needs a definition of the utility under the various possible outcomes.

The critical point here is that the hypotheses need both be defined, and it must be reasonable to assume that they are true! Note that Fisher never was concerned about the "truth" or "falsehood" of the null-hypothesis. But here we must be convinced that either A is a good description of the reality or B is a good description of the reality. Only then can we make a rational decision based on the sampled data. This requires to state the utility (wins or losses) of correctly and wrongly accepting A and B. Based on these utilities one can derive the required confidences for the decisions, what is translated to the "probability of accepting B when A would actually be the good decsription of reality" (-> alpha; type-I error) and "probability of accepting A when B would actually be a good description of reality" (-> beta; type-II error). Sometimes instead of beta the complement 1-beta = power is used.

It seems obvious that this procedure is not suited for research questions. In reaserch it is usually not possible to dinstinguish two precise hypotheses (what should B be, precisely?), and there is usually no chance to state any utility function. But then it is not possible to select alpha and beta (or the power) to justify a rational decision. And if we don't know if the decision is rational, why do we do all this?

A possible way out might be to use Bayesian statistics to get a posterior predictive model and a utility function over the hypothesis space. The decision may then be taken based on the sign of the integral of the product of posterior x utility. This had the advantage that B would not need to be specified (the posterior will tell us what we should believe about B, given the data), and that such decisions would be in a sense rational, but it would again impede the control of error-rates.

Prof Jochen has said it all

There are likelihood functions and even posterior predictive distributions. Both seem to be more "convenient" or "understandable" than p-value functions, and neither of them has been or is used to any relevant degree. They all share the problem that they tell a more complex story but researcher call for simpler black-and-white pictures.