Customarily, the null hypothesis states that there is no relationship between the two variables under the stated conditions.

Now the question is how do we reject the alternative hypothesis in standard testing procedures?

While we perform a hypothesis tests, there are two possible outcomes. The first one is when our p-value is less than or equal to our significance level (0.01 or 0.05), we reject the null hypothesis, H0 and conclude that the data favours the alternative hypothesis, H1. On the contrary, if the p-value is greater than the significance level, then the null hypothesis is not rejected, but the alternative hypothesis H1 gets rejected. Now, in case we do not wish to reject alternative hypothesis we need to alter the p-value. This could be achieved in a number of ways, few are:

1. We can check the multi-collinearity among the independent variable through checking the VIF score and accordingly adjust our number of independent regressors

2. We can also change the sampling techniques and extract a renewed sample for our testing purposes along with its optimum size, if of course this is admissible.

3. We may also check if there are any issues with the normality of the data, especially if we’re performing any parametric tests, etc.

Hello Rafael,

The type of test you envision is conceptually comparable to equivalence (non-inferiority) testing. Since a mean difference between two groups can be evaluated as testing the correlation that r(group membership, score) = 0, I think you could recast this method to your purpose.

Here's a link that offers a simple overview of equivalence testing: Article Equivalence Testing for Psychological Research: A Tutorial

Good luck with your work.

It's a bit weird when you talk about the null and the alternative hypothesis. This makes no sense.

If you talk about a null hypothesis, then there is no alternative hypothesis. The null is either rejected or not. This is a Fisherian significance test (judging the statistical evidence of the observed data against the null) - I'd call it a rejection test.

If you talk about some alternative hypothesis, then there are two alternative hypotheses, A and B, and the task of the test is to accept one of these alternatives with a given confidence - it's an acceptance test or A/B-test after Neyman and Pearson.

It is very important to understand that these are based on very different and incompatible philosophies (although the math behindis almost identical and lead to the unfortunate hybrid NHST approach taught in many statscourses and books).

If you really have two alternatives (A and B), then is does not matter which of these is which (A may be a difference of 0, and B may be a difference of D, or the other way around). You define the confidenceswhith which you aim to accept either of these to alternatives in favour of the other.

If you don't have any alternative to a difference of 0, you can calculate the confidence interval and check if this interval is in a sufficiently small region around 0.

For measuring invariance, you could go for equivalence testing (EQ) as suggested by David Morse. However, you need a large sample for EQ to ensure sufficient power rate. Counsell et al. (2019) examined various methods for EQ and provided some insightful suggestions. Their study is referenced below.

Counsell, A., Cribbie, R. A., & Flora, D. B. (2020). Evaluating equivalence testing methods for measurement invariance. Multivariate Behavioral Research, 55(2), 312–328. https://doi.org/10.1080/00273171.2019.1633617

Article Evaluating Equivalence Testing Methods for Measurement Invariance

If I understand correctly, you probably want to think about band hypotheses in this situation, not a point hypothesis. Then the set up is conceptually pretty similar to the norm, though you need some adjustments, see

@article{MurphyMyors1999,

author = {Kevin R. Murphy and Brett Myors},

year = {1999},

title = {Testing the hypothesis that treatments have negligible effects: Minimum-effect tests in the general linear model},

journal = {Journal of Applied Psychology},

volume = {84},

pages = {234--248},

doi = {10.1037/0021-9010.84.2.234},

}

@article{SerlinLapsley1985,

author = {Ronald A. Serlin and Daniel K. Lapsley},

year = {1985},

title = {Rationality in psychological research: The good-enough principle},

journal = {American Psychologist},

volume = {40},

pages = {73--83},

}

You might also consider the Bayes factors for this non-nil null band.

I agree with the previous proposals and would like to add another one, Bayesian parameter estimation, to get an estimation of the most probable population values for your parameters of interest (under the given priors). For me a very intuitive book to understand the concepts is Kruschke - Doing Bayesian Data Analysis. You are able to define Regions Of Practical Equivalence (ROPE). If the credible interval (e.g. HDI) of your parameter falls within those intervals, you have evidence for either the one or the other hypothesis. But similar to Jochen's description and/or equivalence tests, you need an idea how large effects/differences should be to be of practical relevance, to define the bounds for an A/B test, equivalence bounds or the ROPE in Bayesian parameter estimation.

You asked explicitly for literature in the realm of psychology: search for Daniel Lakens, he published several article about equivalence testing and he is a psychologist, therefore, the examples should be in this direction. Besides that, he as an informative blog with latest news and preprints.

Wow. Thank you very much. That was exactly what I was searching for. Amazing.

Stay healthy!

Rafael