In many studies, people consider results as statistically significant if the p-value of the data analysis is less than the prespecified alpha (significance level) of 0.05. Thus, many researchers (including me!) are often more enthusiastic about in-depth discussions of our results when our test statistics of the relationship between variables or difference among sample groups shows an alpha value of 0.05 or less. Some differences have huge ecological implications at an alpha value of 0.1 or greater. That could also be why we do not always get the threshold for statistical significance we often set at 0.05. Why don't we focus more on the ecological importance of ecological observation than the statistical significance?
This is exactly why people don't always use an alpha of 0.05. David Booth
These concepts are no alternatives and they cannot be substituted or interchanged by each other. The statistical significance is only about your observed data, judged in relaion to some kind of model you use that might explain some aspects of the data, and about nothing else. Particularily not about the sensibility or usefulness of the model.
A high stat. significance indicates only that your observed data provide some sufficient amount of information about an aspect of the model. This aspect is typically a parameter or coefficient value, which often represents a mean change (as a difference between conditions or a "slope" over time, location, concentration, etc.). Using the model, a value of this interesting parameter is estimated from the observed data. The statistical significance is then used to check if the data provide sufficient information to statistically discern this estimate from some postulated or hypothesized value. If -and only if- this is the case, then -and only then- it is acceptable to interpret the estimated value with respect to the hypothesized value.
A simple example: You want to see if the amount of N incorporated by trees depends on the average yearly temperature T (as part of the climatic conditions the tree grows in). You collect data from many trees grown in different climatic conditions. You model the N content as a log-normal distributed random variable of which the expected value depends on T, like E(log(N)) = b0 + b1*T. This is a simple linear regression model of log(N) against T (which may or may not be a sensible model - it requires substative knowledge to judge this!). The coefficient b1 estimates the in the expected log(N) per unit of T (Kelvin, for instance). The model will give you an estimated value for b1 for which your observed data are most likely. This doenot tell you anything about the "quality" of the estimated value. The same data may be not much ess likely for considerably different vaules, and one may also get considerably different estimated values when repeating the experiment (using data from different trees). Now the question is: what can you say about the dependency of N and T (according to your model), given the data you actually observed?
An often interesting question is if you can at least say something about the direction: does the incorporation of N increase or decrease with T? The estimated value has a sign, it is positive or negative. But can you interpret it? To find this out, you can test the data against the hypothesis that b1=0. If this gives a very small p-value, then your obsreved data allows you to statistically discern the estimated value of b1 from this hypothesized value, what means that you can say if b1 can be expected to be positive or negative. Otherwise (no very small p-value) your data don't provide enough information to say if b1 if positive or negative (it's not conclusive regarding the direction of the effect).
Judging the ecological relevance requires some additional statement about what change is considered relevant. Let's denote this minimally relevant change with R. It's straight forward to test your data against this hypothesis: b1=R. If the estimated value of b1 can be statistically discerned from R, then you can conclude if the expected dependency is relevant or not. Usually people don't test such hypotheses directly. Instead they test the continuum of all possible hypotheses and provide the interval of hypotheses that can not be statistically discerned from the estimated value as a confidence interval. You can then quickly see if this interval is outside ofsome "band of irrelevance" (however a scientist or politician will define this).
Jochen Wilhelm , thank you for the excellent explanation. I fully understand your points! Maybe, the way I put my question is misleading as a suggestion to use ecological importance/relevance instead of statistical significant when judging the relationship between variables or difference among sample groups. My point was to provoke a discussion on to what extent we relay on p value (s) of 0.05 or less to say observed data (relationship between variables, difference between groups and/or change in response to a treatment or external stimulus) is relevant. For example, when we do a regression model, slope of a correlation can be significant (e.g., p
Geshere Abdisa Gurmesa, that is where qualitative data comes in handy. If your view is that the ecological aspects are more important than the statistical differences, then design your study to capture the qualitative aspects (ecological descriptions) as well. That way you will be able to capture both magnitude and contextual issues. I think that is the practice for ecological studies.
Assume you make an ecological observation and wish to verify or quantify using data. Characteristics of the observation might indicate to you how to sample for and model your data. You select a test of the model that is relevant to the type of data and further select an alpha for the test that you require to label the data statistically significant.
Collect the data and perform the test. Significance is determined based of the data and the model. Significance does not pertain to your observation. Say you fail to obtain data significance. You conclude that the modeling and sampling failed. You can now conclude observation failure or re-evaluate the model and sampling. Belief in a strong observation indicates a need for a new model and sampling strategy. You may conclude you need more data, different data, or a more appropriate model. ( A more appropriate model is not an invitation to p-hacking.)
There is the opposite outcome of obtaining statistical significance. You cannot automatically conclude that the modeling and sampling were successful. You, still, must re-evaluate the modeling and sampling. Statistical significance is never enough. You must show that the modeling and sampling comport with the observation in a way that your conclusion of relevance is supported. (Note the alpha chosen for significance is the lowest level of the test you will accept. Significance means you have a barely-sufficient result in the proper direction. You might have reached the lowest rung on the ladder of the investigation to support your observation. There is a good chance that you cannot repeat the results.)
Jochen Wilhelm stated in answer to your question "Why don't we focus more on the ecological importance of ecological observation than the statistical significance?" that "These concepts are no[t] alternatives and they cannot be substituted or interchanged by each other." I have attempted to further explain his excellent answer, because you asked again, "So, going back to my question, can we find a balance between statistical significance and ecological relevance to say our observation is relevant in ecological research?"
No. There is no balance.
Geshere Abdisa Gurmesa ,
you say "My point was to provoke a discussion on to what extent we relay on p value (s) of 0.05 or less to say observed data (relationship between variables, difference between groups and/or change in response to a treatment or external stimulus) is relevant." -- such a discussion is nonsense. The p-value is not about relevance whatsoever. There is nothing to discuss.
This would be like discussing to what extend we relay on the size of a wheel of a car to say that the car will drive us to nice places or in some relevant direction. This would be a stupid discussion. Is a neccesary condition for a car to be able to drive to have sufficiently sized wheels. After having asserted that, it's a different question to ask in what direction or how fast we believe the car actually moves and if this may get us to a nice place.
Instead of discussing such things we should focus on the correct meanings of terms like "data", "model", "effect", and "p-value" and we should fight confusions and gobbledygook.
Joseph L Alvarez thank you! Your explanation contributes to my understanding of statistical tests and its interpretation. I know why we do statistical tests. I found some good sense of good answer to my question in your statement "You may conclude you need more data, different data, or a more appropriate model. ( A more appropriate model is not an invitation to p-hacking.)".
Jochen Wilhelm , I am here to learn! Wouldn't it be more professional if you read the whole points I raised in that paragraph to understand the dilemma I sometimes have when using, instead of picking a sentence and labelling it as 'nonsense"? Assume that you are asked this question by one of your student, is it how you answer? If "The p-value is not about relevance whatsoever', which I also believe so, then explain to me how should I separately treat them. For me ecological relevance of an observation is very crucial. Help me understand how can I use statistics to detect those 'ecologically relevant observation using statistics (if possible). Note that your help here is purely a volunteer work. If you are not patient enough with back and forth questions (or you have "nothing to discuss") from someone who wants to have a practical understanding of statistics, just safely ignore it!
Geshere Abdisa Gurmesa , it seems you felt offended by my answer. That was not intended, and I apologize for having given that impression. I try to be clear and direct, and I still think that my answer covers all aspects you re-raise in your last comment. Maybe you try reading my post again in "offence-mode off" and assume that this is not written by impatience to to upset you in any way. Otherwise, well, you may safely ignore it :)
Maybe one again: Judging the ecological relevance of an observation has nothing to do with statistics. These are different worlds, and you should not confuse them and not even bring them in contact. Statistics is used to judge your data relation to a (statistical) model. It allows you to quantify the amount of information your data provide to estimate a coefficient in that model, relative to some hypothesized value. This hypothesized value can be any arbitrary value; if this value is ecologically relevant or not is nothing that could be answered with statistical tools.
Say you collect data to estimate a mean difference (between two groups) D. You believe or assert or know that D > r would be ecologically relevant. Using your collected data, your best estimate of D is d. This d might happen to be smaller or larger than r. Since there is uncertainty involved, it may happen to be larger than r even when the (unknown!) D < r, and vice versa. But D is unknown. You know d, but you don't know how much information about D it gives! The statistical test of the hypothesis D = r can now be used to judge if the information in your data is sufficient to discern d from r. If this is the case (i.e. when you can reject this hypothesis), then the conclusion D < r (i.e. the effect is ecologically irrelevant) is warranted when d r (the effect is ecologically relevant) is warranted when d > r. If this is not the case (i.e. you can not reject the hypothesis), you can not make aclaimabout the relevance of an effect because your data did not contain enough information about this in the estimate of D (the data are inclonclusive w.r.t. to D = r). Instead of testing a particular value D = r usually confidence intervals (CIs) are constructed that contain all values of D for which the corresponding hypothesis can not be rejected. One can then simply check if the CI is completely above r (in which case the conclusion is warranted that D > r, so the effect is ecologically relevant). If the CI is completely on the wrong side of r, the conclusion is warranted that D < r, so the effect is ecologically irrelevant). If the CI includes r, the data are inconclusive and neither of the conclusions is warranted.
Jochen Wilhelm , thank you very much! I re-read your previous and latest answers "offence-mode off" :) Your latest explanation cleared the confusing ambiguities in my mind. Please, note the confusion I had contributed to the loss of temper in my response given how important the question was to my understanding (because I do not believe there is 'nonsense' question, especially when asked to gain practical knowledge). I work with field data, which is sometimes derived from few replicates. The p-value, the test statistics, that I and my colleagues use, sometimes show insignificant mean difference among treatments, even though the observed difference can have import ecological relevance. I face the challenge to tell my stories when writing because I know many people in this business are less impressed by nonsignificant changes. This was why I brought the question on this platform. Thank you for asserting that judging the ecological relevance of an observation has nothing to do with statistics.
Hi Geshere Abdisa Gurmesa ,
It is interesting this paper about "Statistical Significance and Biological Relevance" https://www.efsa.europa.eu/en/efsajournal/pub/2372
Any other decision about "relevance" could be set by a political choice (not by scientists)
Geshere Abdisa Gurmesa There is a difference between an ecological observation and observation of an ecological effect. There is a difference in assigning importance between the two. An observation is important when is well-documented and comprehensive. (Somebody has to count the butterflies.) Observation of an effect is important when it is documented and comprehensive. Cause is not necessary to observe an effect. Documentation of an observation or observation of an effect includes some statistics depending upon the nature of the report. Documentation of an effect includes a description of the effect and its magnitude ( larger, smaller, increasing, decreasing, ...). An effect (change) that cannot be quantified is an observation, not the observation of an effect. The change (new observation) is important because it contributes to the knowledge surrounding a possible effect.
It is difficult to sort or qualify importance when a requirement for publication is statistical significance. It is necessary that we clearly describe the importance of an observation, instead of bastardizing the report to claim statistical significance.
If an investigation is intended to document an observed effect, the statistics should be performed before designing the investigation. Assume a data distribution, select the model, decide the test significance value. The significance value is the minimum level for accepting the effect is observable. The effect is not quantifiable at the minimum observation level, so, it is advised that a quantifiable level be defined for data collection beyond minimum observation.
Data analysis, after completion of data collection, is performed as data. This analysis has nothing to do with importance or the effect investigated. It is a measure of data against the model and the strength of the data for quantification.
The terms significance and hypothesis testing tend to be conflated with effect and effect size. This conflation is unfortunate. The statistical analysis is independent of the intent of the investigation. You decide and defend the amount of support the statistical results provide the intent.
Joseph L Alvarez Thank you! It is true that the terms significance and hypothesis testing tend to be conflated with effect and effect size, and I think that is what I faced. Now, you have explained it in a plain language.