Your statement as such makes sense (then and still). That the difference between the trends is not significant is your judgement. However, if this judgement was done by the rule "since p>0.05 the difference is not significant" without considering the context (effect size, experimental conditions, scientific value, plausibility, etc) the decision might not have been very sensible or rational (in a way). But the statement itself is ok: you do not consider the difference significant, that means: your experiment did not provide enough data to see any practically relevant difference between these trends to be worth a further investigation of careful discussion. You say you have no reason to distinguish these trends based on your possibilities of getting data (what is limited w.r.t. to amount and precision). This is your decision, based on your expertise in the domain, and (hopefully) supported by plausibility.

You can not test the equality of µ1=µ2. You can test how incompatible your data is with a statistical model in which µ1=µ2. The question is not "whether or not", it is about "how much". This yould be a significance test. This "how much" is measured by the p-value, and a judgement about what to conclude from this value depends on the context.

If you want to do a hypothesis test you compare two statistical models, or one model with a particular parameter set to two different values, according to two different substantial hypotheses. So you may formulate hypothesis A as µ1=µ2 and you must also formulate an alternative B as µ1=µ2+d, with d having some defined value that makes sense in the calculation of the utility or loss. After going through all these (in reasearch often impossible) steps of defining a rational utility/loss function and a rational selection of alpha and beta, and the determination of the required sample size, you gather the data and subject it to the test. The test result will be either "accept A" (meaning: "act as if A was a good describption/model of the world") or "accept B" (meaining: "act as if B was a good describption/model of the world"). There is no questiono if A or B is "correct", and there is no answer given to this question. The only thing is that if A, B, alpha and beta are chosen on some rational grounds, than the decision strategy will be rational (w.r.t. the criteria used to fix alpha and beta).

See the answers to the questions below.  If you use the alpha and beta values you want in determining sample sizes you should be fine. Software like G*power should be helpful. Best wishes.

https://www.researchgate.net/post/Are_we_misusing_the_P-value_statistics_in_diversity_studies_What_options_do_we_have_to_implement_the_hypothesis_testing

https://www.researchgate.net/post/How_to_Determine_Sample_Size_for_Diabetic_and_CKD_Population

You ask for hypothesis tests. Thats actually one of the major points that got missed and confused when using / interpreting p-values. p-values are a sample statistic that is used for significance tests, but it has no essential meaning in hypothesis tests. As long as this is not clearly understood, the discussion will remain blurry and misleading.

A hypothesis test REQUIRES to define alpha and beta. Without their definition a hypothesis test is impossible. Whatever is done is not a hypothesis test, and it is illogical and wrong to talk about error rates in these cases.

The next problem is that the decision based on hypothesis tests are as ratioal as the selection of alpha and beta. It is not clear in research how to select rational values of alpha and beta, so it is actually not clear how to use a hypothesis test to make a rational decision. I find it kind of funny to adhere to some complicated formalism with all those exact calculations to arrive implement a decision rule from which we don't know if it is rational (as long as we just use some "common" value for alpha and beta).

There is no free lunch in nature. There is no way of an automatic definition of what results are "significant"/"relevant". These are judgement calls. They must be made under consideration of the experimental and scientific context and based on appropriate expertise. There is no assurance of correctness and reliability. All we have is plausibility.

--

Just one more comment on your statement "It is now clear that p-value can not be used to determine if an hypothesis is true (or if results are important)...":

Hypothesis tests are not to declare hypotheses are true or false. They aim to provide a rule for (possibly) rational decisionos between to substantially different alternatives. It is not relevant if the hypothesis is true or false. It only matters if the consequences of our decision will maximize the expected utility (or minimize the expected loss).

Significance tests are neither about the truthness or falsehood of H0. They aim to assess the "incompatibility" of the data with H0, without making any statement about the truthness of H0. It is clear that H0 is just a mathematical, abstract construct needed as a "leverage point". Nothing else. Rejecting H0 does not mean to "proof" that H0 is false. H0 *is* false, anyway. A significance test is a tool to help us seeing whether our data is already sufficien to "see this clearly enough". Rejecting H0 has nothing to do with error-rates or with alpha (the only "error" that can happen is to be unable to reject H0, what is a false negative! False rejections are not possible). Therefore it is rather silly to define a strict decision rule like "reject H0 whenever p

Thank you, both of you for your clarifications.

To make myself clear, one day I wrote :

"The immune responses to L.plantarum, showed the same trend as L. zeae and B. bifidum, but differences between the three drying treatments and the control were not significant (p-value = 0.28)."

Telling that does still make sense without telling the beta of the same test ?

Or considering the p-value and the results (means of each group) I can tell this assumption because of my "expertise" in the domain ?

Finally, how do I test if an equality (µ1=µ2) is significant or not ?

Your statement as such makes sense (then and still). That the difference between the trends is not significant is your judgement. However, if this judgement was done by the rule "since p>0.05 the difference is not significant" without considering the context (effect size, experimental conditions, scientific value, plausibility, etc) the decision might not have been very sensible or rational (in a way). But the statement itself is ok: you do not consider the difference significant, that means: your experiment did not provide enough data to see any practically relevant difference between these trends to be worth a further investigation of careful discussion. You say you have no reason to distinguish these trends based on your possibilities of getting data (what is limited w.r.t. to amount and precision). This is your decision, based on your expertise in the domain, and (hopefully) supported by plausibility.

You can not test the equality of µ1=µ2. You can test how incompatible your data is with a statistical model in which µ1=µ2. The question is not "whether or not", it is about "how much". This yould be a significance test. This "how much" is measured by the p-value, and a judgement about what to conclude from this value depends on the context.

If you want to do a hypothesis test you compare two statistical models, or one model with a particular parameter set to two different values, according to two different substantial hypotheses. So you may formulate hypothesis A as µ1=µ2 and you must also formulate an alternative B as µ1=µ2+d, with d having some defined value that makes sense in the calculation of the utility or loss. After going through all these (in reasearch often impossible) steps of defining a rational utility/loss function and a rational selection of alpha and beta, and the determination of the required sample size, you gather the data and subject it to the test. The test result will be either "accept A" (meaning: "act as if A was a good describption/model of the world") or "accept B" (meaining: "act as if B was a good describption/model of the world"). There is no questiono if A or B is "correct", and there is no answer given to this question. The only thing is that if A, B, alpha and beta are chosen on some rational grounds, than the decision strategy will be rational (w.r.t. the criteria used to fix alpha and beta).

All right, everything is clear now.

Thank you ! :-)

Cyril -

Two more points:

If you want to "test" the difference in two means, I would give a confidence interval (or maybe more than one confidence interval) around the difference in the estimated means.  As with a p-value, a confidence interval is driven by sample size, but I think it gives you a much more practically  interpretable result. 

Also, you noted that "1 - beta = power of the test."  That takes me back many years to when I worked with hypothesis tests for very small sample sizes.  In such a case, the null and alternative hypotheses can be examined by looking at histograms showing probabilities for each of a finite number of possible events.  If you look at the distribution for an alternative hypothesis, you cumulate the probabilities for events lower or higher than the observed event, depending upon whether you are estimating the probability of being (hopefully) in a tail of that distribution (to estimate beta), or the remainder of that probability distribution (to estimate power).  But when the number of such discrete possible events is relatively small, it matters if you count the observed event for both 'calculating' beta and 'calculating' power.  As I recall, many people, to make 1-beta=power, would not count the observed event in the power 'calculation,' but to me it is more logical to count it in both cases, and thus power + beta > 1.  But hopefully just a little greater than 1 or you really do have such a small sample that large effects are difficult to see.  It would be analogous to wide confidence intervals.  Anyway, I just thought you might want to be aware of this issue.

Cheers - Jim 

PS - A third 'thing': The short attachment might also  be useful regarding understanding the nature of hypothesis tests.

 https://www.researchgate.net/publication/262971440_Practical_Interpretation_of_Hypothesis_Tests_-_letter_to_the_editor_-_TAS

Article Practical Interpretation of Hypothesis Tests - letter to the...

Thank you very much James !

You just gave me useful information.

Your paper deal exactly with issues we have in applied microbiology : small sample size (because of cost, material and time) and high request for small alpha...

Regards