Dear Fabio,

    There is no real consensus about the best solutions; statisticians tend to believe that certain answers are best considered within a particular context. Replacing logic with rigid and formulaic interpretations of (often) misapplied models is more or less why they feel the warning is needed.

   Most papers that we read do not reflect this reality. So please do not take the existing literature as a guide to what would be the best.

   If your research question is about diversity, then your best answer must be some representation of this. It is your choice whether this is something like a concentration or richness parameter, or some number of species observed, etc... Now, imagine that you use an ANOVA - on some assumptions about the distributions of said observations. At best this tells you, to the extent that the model fits, whether the average numbers under each conditions differ by more than would be expected by chance. It does not reflect all of the possible parameters you could have estimated (nor all of the hypotheses you could have tested).

   The p value associated with this model you have decided was relevant does not describe the size of the difference in any simple way. So statisticians would suggest that you examine the estimated size of the difference of interest as the benchmark of its importance. And that you make sure your model corresponds to your knowledge of the situation. Then they would point out that in many situations, the p value you obtain can be used to make a decision - which will be correct, under the model, with a certain frequency. Unfortunately, this is not the same as saying that the p value actually reflects the strength of the evidence, the accuracy of your parameter estimates, or even whether you have wisely chosen a model. It is these sort of inferences which should NOT be made based on p values.

   Options? Zuur et al wrote some nice books about analyzing ecological data. Regardless of the sources, just remember that the descriptive function of statistics is best served by a meaningful statistic. So consider how you would best describe what you wish to observe and find a statistic that corresponds to this. And remember that the model of chance may not have any importance to your situation at all. For example, are you really hoping to generalize your findings to forest far away? or are you really trying just to describe the differences you observe between the situations you study. The latter does not require any inference at all.

Hope this helps.

This is a good question and I thank you for caring enough about your research to ask it. There are a couple of issues here. One is that statistical tests require certain things to be true before you can put much faith in the results.  For example, a 1-way ANOVA F requires normal populations with constant variance etc usually tested on the residuals. If you haven't verified these things you have no right to believe the probability statements(e.g. p-values). Note there are sometimes ways to fix things up if one or more assumptions are violated. If you can't then you need some other test whose assumptions you can verify. Another issue is always seeing that if p

"So I'm wondering if really testing diferences of for example, species richness between factors (e.g., Forest edge or interior) with an ANOVA test can't give us answear of hypothesis testing? and which additional measure are desireable to test in this type of approach?"

 It can give if properly used and interpreted.

Additional measures are confidence intervals, effect size estimates, power and sample size clarifications.

There are other threads which discuss a similar topic like the following

https://www.researchgate.net/post/What_are_the_alternatives_for_p-values

https://www.researchgate.net/post/Is_rejecting_the_null_hypothesis_because_of_p005_valid_statistically_or_clinically

https://www.researchgate.net/post/When_the_significant_differences_P_value_less_005_should_be_consider_as_clinical_important_seriously

Just a short comment on your concern about the "robustness of p-values":

"[...] stating that p-value was robust or significant. Even I also have stated that type of assumption. And now, according to the warning that ASA gave, this isn't robust..."

You must make yourself clear that the "p-value" is a random variable, just like the obserevd data is from a random variable. P-values are calculated from your random data just in a way that ensures (given some assumptions hold) that they will have a uniform distribution over the unit interval under the null hypothesis. This is all we know about a p-value used in a significance test: that its distribution under H0 is uniform. If H0 is not true, the distribution will be right-skewed, ideally L-shaped, showing a high density close to zero and a low density close to 1. The precise distribution is not known to us. It is related to the "power": the more sensitive the test, the larger the effect size, and the larger the sample size, the more L-shaped will the distribution of the p-values be.

What we actually want to know is if this distribution is uniform or if it is skewed.

Now here is our problem: We do not know this distribution. We have only a single p-value out of such an unknown distribution, and now we want to infer how this distribution might look like. This is impossible. All we can say is that it is more likely to observe a small p-value when the distribution was skewed. This means, it will happen more frequently to get rather small p-values from skewed distributions than from uniform distributions, and that the probability to get p

Fabio -

As a member of the ASA, I replied to an ASA thread on that press release expressing that it was past due.  I have never reported a p-value without other information.  The type II error analyses, or similar, that people seem to have rediscovered recently have always been essential to decision making.  However, this seems to have been all but forgotten at least forty years ago. 

 https://www.researchgate.net/publication/262971440_Practical_Interpretation_of_Hypothesis_Tests_-_letter_to_the_editor_-_TAS

I think two phenomena have contributed to this problem: (1) My fellow mathematical statisticians have often gone into great depth to study null hypotheses as mathematically interesting, and forgotten about practical use.  (2) Many people in other fields who count on statisticians to give them useful tools have latched on to the idea that the p-value is such a tool (it isn't), and without proper emphasis on dissuasion from statisticians, this has become a huge problem. 

I used hypothesis tests years ago, and even developed a new one for a specific purpose.  Done carefully, an hypothesis test (not just a p-value) can be useful, but actually I have not felt the need to use one in many years.  (I did end up at odds again with other statisticians' views since then because I used regression for estimating official statistics for highly skewed establishment surveys, with a type of sampling that has not been popular because almost no one else has considered the big picture of 'total survey error' in such a case.  It worked very well.  But again, the statistical 'establishment' can be very dogmatic and not emphasize what is practical.) 

I am not familiar with your subject matter, but as Bruce said, the "...function of statistics is best served by a meaningful statistic."  An isolated p-value is something like giving you one end of a confidence interval with no clue as to where the other end might be. 

Perhaps the most important thing you might do would be to try to validate your results.  Scientists validating each others results should not be considered a bad reflection on each other, but as a useful statistical tool. 

You could study effect size, but the "small, "medium," and "large" categories may be misleading. 

As I noted, I do not know your subject matter, but if a confidence interval or a prediction interval may be used, that would be meaningful, useful, and easy to interpret for all practical purposes (though perhaps most may not exactly interpret them technically correctly).

Best wishes - Jim

Article Practical Interpretation of Hypothesis Tests - letter to the...