No.

A t-test (or any hypothesis test in general) is unable to guide you whether or not there is a difference. Such tests only tell you if your sample size is large enough to interpret the estimated sign of the effect. It answers a test regarding your sample size - not regarding the effect!

Interpreting a result with a large p-value as "absence of a difference" or "absence of an effect" is logically wrong. See:

https://www.bmj.com/content/311/7003/485.long

To show that the tests produce equivalent results, calculate the confidence interval (CI) of the difference between the test results and check that the limits are all within the equivalence band.

For instance, say that a difference of less than 0.1 would be considered still equivalent. Then the equivalence band is from -0.1 to 0.1. If your CI is from -0.2 to -0.05, you can not conclude equvalence. If the CI if from -0.08 to -0.02, then you can conclude equivalence (although a p-value for zero defference would be significant!). If the CI is from -3 to 1.5, you can not conclude equvalence (although the p-value for zero difference is almost 1!).

This procedure is also known as TOST (two one-sided t-tests).

I hope you get the point.

There are a few other statistics that may be useful. One category are statistics of "accuracy" like MAE and RMSE, and others. ... I think plotting the data and superimposing a 1:1 line gives a good sense of how similar the results are (and an indication of over what range you tested the procedures). ... You might also fit a (linear ?) model, and a) look at the r-square, but also b) it may be the case that the results of the new method are better represented with a regression to the old method.

Rather than testing a point null hypothesis, you could define what you meant by "similar" thereby creating an interval to test. Some discussion of that is here: Article Rationality in Psychological Research: The Good-Enough Principle

Thank you all for your replies, I appreciate it!! Sal Mangiafico Daniel Wright Jochen Wilhelm

Jochen Wilhelm I have a question. If I do the Cohen's d of two variables whose difference should be almost 0, I'll never get a Cohen's d larger enough to include the CI and so my TOST will be not-significant. What am I doing wrong?

Do not use t-test.

Use One-Way ANOVA and after Measure the "difference (in the means)".

IF you want you can send me your data and I' will show ...

Massimo Sivo aren't the oneway ANOVA for two samples and a two-sample t-test the same? The t^2 teststatistic of the latter is the same as the F-statistic of the former, so how does this help? And how does this help to assess equivalence, which is not possible with this approach? Please clarify.

Absolutely NOT....

If you have 5 measures with method A and 5 measures with method B, and you use Anova versus t-paired test you get different df (degrees of freedom) for the residual...

Rainer, t^2 is the same as F ONLY WHEN they have the same number of dfs

Massimo, it is simply wrong what you are say, because you are referring to two completely different models. You are comparing apples with pears.

Typically, "oneway ANOVA" refers to a comparison of independent groups, therefore in the two groups/samples case, it is identical to the "independent t-test". If you have repeated measures, you would use a "repeated measures ANOVA" (which I have never seen to be labeled oneway ANOVA, but please correct me). In the two measure times case, this "rmANOVA" is again identical to a "dependent/paired t-test".

Of course, if you compare the results of a oneway ANOVA with a paired t-test, they will show different results, but this is itself complete nonsense to do so.

And you did not tell us how to check for equivalence with this method, even if your appraoch would make any sense.

While waiting for the data of Federico Roggio, I found two sets of Real data and I went on “comparing apples with pears”:

• · 5 items were measured with method A and with method B.

Here are the data:

Method A: 6.42, 7.15, 7.05, 7.31, 7.16

Method B: 6.73, 6.74, 6.58, 6.77, 6,48

Using ANOVA and t-paired test (rmAnova), on the above data, one finds two opposite decisions, with a 90% Confidence Level.

I would like to ask to the “very High Scored” to prove THEORETICALLY THAT

• · “””A t-test (or any hypothesis test in general) is unable to guide you whether or not there is a difference. “””

and

• · That it is wiser to lose 50% of the degrees of freedom, by using the t-paired test (rmAnova)

The method I propose is able to prove the equivalence IF the DATA tells US that the equivalence “EXISTS with a given CL”.

Rainer your statement “And you did not tell us how to check for equivalence with this method, even if your appraoch would make any sense.” arises, in my opinion, from the NON-use of the Theory…

Massimo Sivo thank you for your reply. What kind of data do you need? The whole set or just means, sd, etc?

Massimo why would you compare the results of an test for independent data with that from a test for dependent data? They are supposed for different data structures and somewhat mutually exclusive, either you have dependent data or you have independent data. Yor example, show us, how you compare independent data with a dependent t-test. How will you match/map the values to each other, if there is no inherent connection between the samples (therefore "independent"!). Do you make it up? Do you even know what the problem is?

Tell us your theory and let us know about your genius ideas, I believe it is as flawed as your use of statistical tests. You never proved anything here or in your pamphlets, empty words and shallow promises. Nothing that you ever demonstrated. Poor.

Federico Roggio , I just saw your question from Dec 2nd. I did not understand it. Could you please elaborate / give an example?

Federico Roggio, send the complete data to see what I can do...

Jochen Wilhelm I though that I had to select as upper and lower limit the Cohen's d ranges, which was almost impossible to state equivalences (since my d = - 0.015). Another researcher explained me that I have to choose the limits my byself.

Massimo Sivo

Federico Roggio the limits for the equivalence test should be meaningful in your research area. What is the smalles effect size that is of practical relevance for your field, how large has an effect to be, i.e. a difference between your groups, so that it matters? Nobody from outside your field can answer this question, therefore, it is necessary that you determine it for yourself. Your empirical d = 0.015 is "small" in a lot of fields, but maybe this would be a meaningful improvement in your field, so that your Smallest Effect Size Of Interest (SESOI) could be smaller than that. You can search for SESOI to find articles concerned with it.

Yes Rainer Duesing I studied the SESOI thing and all the others related to TOST. I appreciate your support.

You should possibly also consider a mean-difference (aka Bland-Altman) plot. The median difference (test2-test1) is -0.75, the IQR of the difference is (-1.3, 0.015). The interdecile range is from -1.86 to 1.11. There is a subset of 3 points a bit higher than the rest. These points reduce the bias but increase the variance. The differences excluding these points pretty well follow a normal distribution, as you can see in a normal-QQ plot. You might cross-check why these points are so "large" in test 2 or so "small" in test 1. If you find a good (plausible) reason what could have gone wrong there, the rest then shows good accordance but a considerable bias (test 2 < test 1).

Rainer Duesing

As you can see ( in the data I used) the correlation is 0, with a confidence level of 90%.

• I ask you again “why lose degrees of freedom”?

SIGNATURE:

The person who ”…never proved anything here or in ... pamphlets, empty words, and shallow promises. Nothing that you ever demonstrated. Poor.”

Federico Roggio

Using your data the two methods are NOT equivalent with a Confidence Level of 90%, BOTH with the t-paired test and my way of doing (ANOVA): the two means are significantly different.

I do not understand your points:

• · I did the exact measurement in the same group with two different methods. I want to show that using method 2 (faster and simple), we get almost the same parameters as method 1 (slow and hard to reproduce).
• · Can I state this condition by doing a paired t-test and then argue as a positive condition the p = 0.8?

What parameters are you considering?

From where did you find p = 0.8?

Dear Massimo,

you are right in so far that IF the two measures are really uncorrelated that the sacrifice of df is detrimental for the power. In such a case, we would not consider the measures as "dependent" so that a "dependent/paires t-test" is of not much use.

BUT such a case is very(!!) unlikely in a real world application, as Federico Roggio described. He talked about people who are tested twice, with similar material. Show me a repeated measure from the same enitities, for the same material, within a short range of time, which are totally uncorrelated. People are not random number generators. Therefore you can absolutely expect a correlation between the two measures (which is really high in the sample data by Federico!) and THEN the dependent t-test/rmANOVA and loss of df totally pays off! The loss of df is compensated even by moderate or small correlations between the measures and do not need to be as high as in the sampe data. So, why using independent tests like oneway ANOVA/independent t-test if you can have more power if the samples are correlated?

And you still did not provide any evidence for the test of equivalence...... still waiting

Rainer Duesing

When the correlation is “significant” there is no advantage of ANOVA versus t-paired test.

But, when the correlation is “NOT_significant” there is an advantage of ANOVA versus t-paired test (due to the dofs)

Massimo Sivo yes, I noticed now that R automatically used Welch Two Sample t-test rather than the paired one. the 0.8 was referred to Welch's one

Dear Massimo,

since you love to come up with artificial examples, here is one. Two samples sampled from a normal distribution each. Their empirical correlation is small (but they are correlated) and not(!) significant, but the paired t-test indicates a significant difference, whereas the independent t-test indicates no difference (although I would argue that this really does not matter, if you look at the p-values, but if you take the 95% CI, as it is usually done, the decisions differ)

Still, show us how to establish equivalence.

Best, Rainer

"The tests are not different" is vague, and just because two samples have the same mean does not mean that the agreement is good.

There are two questions to ask about the agreement. The first is ordinal agreement : are the participants ranked in the same order on both occasions?

However, it may be important that the participants get the same absolute value on each occasion, in which case a Bland and Altman plot and limits of agreement statistic is helpful.

You may find it useful to run a type I intraclass correlation. This would allow you not just to look at the reliability of a single measurement but also to evaluate the reliability of using the mean of two measurements.

Dear Professor Ronán Michael Conroy, thank you for your reply. We measured the participants with the two different tests on the same day, with a pause of 1 hour. Two researchers were guiding the participants; one was accomplishing the test-1 while the other was completing the data collection with the test-2.

I think that ICC is not needed since there is not an error made by the researchers because the two tests (doing a specific movement with two different tools) were performed just by the participant, the researcher provided no external support.

Obviously, correct me if I am wrong. I'm just a PhD student!

Aha – if the tools are different, then you are testing the agreement between tools, not between the researchers (though it's a pity that there was one researcher to each tool). This is the classic Bland and Altman design – agreement between two measures neither of which is the gold standard. In this case, you have a two way mixed effects ICC, with the tool as a fixed effect and the participant as a random effect.

Ronán Michael Conroy yes you got it! I'm trying to show that the "tool2" (which just another way to execute a sport test) is comparable to the classic "tool1".

Hello Federico Roggio. For the situation you describe, I like showing a scatter-plot with both the least squares regression line and the line of equivalence (X=Y); and I like to report the ICC, Pearson r, and the mean difference (each with its 95% CI). I think this provides all of the information most folks are likely to want. You can see an example in this Medstats thread:

https://groups.google.com/g/medstats/c/YR-2uLLacrc/m/NeAfgisZAgAJ

HTH.

If all participants used "tool1" first and then "tool2", then the factor time is confounded with the factor tool. In this case you would need to show or at least to plausibly argue why having experience with "tool1" would not impact the outcome when using "tool2". This applies also for showing eqvivalence of the tools: a positive tarining effect may cancel out a negative tool effect (tool2 is actually worse than tool1, but this is compensated because the test persons have experience after using tool1 that helps when using tool2).

Rainer Duesing

Dear Rainer, nice example!

Correlation “NOT_significant” and different conclusions, CHOOSING alfa=0.05

Correlation “NOT_significant” and identical conclusions, CHOOSING alfa=0.07

• Moreover, the SEQUENCE of measures is Out Of Control, both for the Mean_measures and for the Mobile_Mean_differences.

THEN, methods A and B are not good methods: they depend on the SEQUENCE!!!

Federico Roggio

Dear Federico,

• The SEQUENCE of your measures is Out Of Control, both for the Mean_measures and for the Mobile_Mean_differences.
• THEN, the methods Test1 and Test2 are not good methods: they depend on the SEQUENCE!!!

Jochen Wilhelm, I stated it already. The rest time is enough (as previously reported by other authors) to perform the test without any possible interactions of the previous one.

Massimo Sivo what are you saying?

I did two tests with a pause of 1 hour in 40 subjects and you state that the tests are not good methods? lol

Jochen Wilhelm 's point is key here: because the tools were used in a fixed order, performance on test 2 will be affected by performance on test 1 – the person learns by doing test 1 and this learning will be applicable at test 2. If you had presented the tests in a random order, this learning effect would affect each test, and you could measure the effect for each test. However, with the tests administered in a fixed order, you cannot disentangle the effect of prior experience of testing from the effect of the difference in tests.

This is what we call in gaelic ciall ceannaithe – wisdom that you paid dearly for!

Federico Roggio

Dear Federico,

• · Believe it or not, the truth does not change.

Your measurement process is Out Of Control, both for the Mean_measures and for the Mobile_Mean_differences, because they depend on the SEQUENCE…

Analyse your data.

In my opinion, the two methods Test1 and Test2 have both some problems:

• they depend on the SEQUENCE in the measurement process!!!

Ronán Michael Conroy Jochen Wilhelm Both protocols were explained before they started them. So even before the start, they knew already what to do. In this specific condition (I did not explain all the research design because obviously, I am going to publish it), the first execution will not affect the second one because the only variable of interest is the heart rate. I doubt that it can be modified even if you know the exercises you will do. Do you agree with me ?

No you still have a sequence problem and you cannot disentangle the contributions of the condition and the order. If I understood you correctly, condition 1 was with a long time to solve the test and condition 2 with a short time to solve the test, correct? Imagine the following possible outcome: Because of the shorter time to solve the test, the participants feel more challanged and therefore the heart rate will increase. On the other hand, because they already knew in principle how the test will work and get used to the whole experiment, they will generally calm down. Now, as in your data, you did not find an effect from test1 to test2, but you cannot show why. Was the challenge not high enough to increase the heart rate or was the calming effect strong enough to counter regulate the increase. If you do not randomly assing the conditions, you cannot rule out such sequence effects.

When you are in a new test situation your heart will beat faster than normal. When you encounter the test situation a second time, you will be less nervous. This alone is a reson to believe that the sequence in fact can have an effect in your experiment. No matter if you handle tools, answer questions, listen to stories, solve math riddles,...

Rainer Duesing nope man, that's wrong. They have to perform a cardiorespiratory fitness with a x repetition in x time. Nothing beside the HR is going to change. No tests or similar to solve, no things were "knowing already the procedure" could change the outcome, since other authors observed it already. By the way, in sub-maximal tests, HR cannot be "modulated" through your mind (like it happens in different contexts).

I give you an example:

If I was testing the 6 minutes walking test, participants might know that they hit a better score if they walk faster. In this context, your assumption is met.

If we tell them " you have to do 30 jumps (30 centimeters) in 60 seconds" and then, after one hour of rest we say "now you have to do 10 jumps (30 centimeters) in 20 seconds". If the frequency of movement is known, the observed variable may change while the others are already fixed by me.

Jochen Wilhelm Yeah, heart may beat faster, that's why we did not evaluate it alone. Anyway, could you tell me where can I study that "sequence" condition (I can't find a real explanation through google)? It may be worth knowing for future designs.

You can look for "cross-over design".

Federico Roggio

You ask to Jochen Wilhelm

• · Anyway, could you tell me where can I study that "sequence" condition (I can't find a real explanation through google)? It may be worth knowing for future designs.

I told you that

• · Your measurement process is Out Of Control, both for the Mean_measures and for the Mobile_Mean_differences, because they depend on the SEQUENCE…

But you wrote: lol!

• Study the Control Charts Theory!

Jochen Wilhelm Thank you.

Massimo Sivo Probably, you have difficulties translating your thoughts into your fingers typing.

Now, writing "Study the Control Charts Theory!" you asserted a possible way to investigate your thoughts, so thank you.

When you write, "Believe it or not, the truth does not change. Your measurement process is Out Of Control, both for the Mean_measures and for the Mobile_Mean_differences, because they depend on the SEQUENCE…Analyse your data. In my opinion, the two methods Test1 and Test2 have both some problems: they depend on the SEQUENCE in the measurement process!!"

you are just acting like a fool donkey, not a researcher.

Federico Roggio

The fool donkey could find many things (you did not find) about your problem (DATA):

• · I did the exact measurement in the same group with two different methods. I want to show that using method 2 (faster and simple), we get almost the same parameters as method 1 (slow and hard to reproduce).

while you found by the

• · R automatically used Welch Two Sample t-test …. p-value 0.8

from which you hoped that TEST1 and TEST2 be “equivalent”!

• · They are not “equivalent”… and the SEQUENCE is much more significant than the difference in the two methods!!!

Good luck from the fool donkey….

Massimo Sivo Cool, great! You keep acting like a fool donkey anyway. Keep going, your attitude will help you to reach higher positions.

Bye man

Federico Roggio

Good recommended researcher, WHO did not find the significance of factors, using R, commenting Scientifically…

Good luck from the fool donkey…., again

Massimo Sivo, I still have a lot to learn, my dear, that's the problem heheh!

Anyway, thank you for your suggestion, even if we have different points of view, you helped me to improve my skills.

Buon Natale Massimo!

Thanks for the interesting comments և discussions.

Best regards