Hi Achim,

That is an interesting question and I think there are many ways to answer it. One of the most straightforward probably is to use 1 - P-value for each statistical test assuming H0: meanGroup1 - meanGroup2 = 0.

However, I would rather 1) use the coefficient of variation (cv = sd / mean) to filter noisy variables and subsequently 2) minimize a quantity of the form

meanVarianceWithinGroup / meanVarianceWithinSample

with

meanVarianceWithinSample = (var(sample1) + ... var(samplem)) / m

meanVarianceWithinGroup = (var(group1) + ... var(groupn)) / n

assuming you have m replicated samples across n conditions. This in turn will give you the profiles that change the least with respect to treatment having the average intragroup variance as a reference.

If you want to derive P-values in this 2-step framework you can always try simulations or permutations.

I hope someone will comment on this!

Confidence intervals for differences between means, proportions etc are indeed calculable. Rather than saying how confident you are that the values are not changing, however, I'd say that you need to quantify the likely size of the change. If the confidence intervals for the change cover a clinically insignificant difference, then you can report that whatever change exists is not important.

What you are looking for are equivalence hypothesis test. The idea is you define the smallest difference you deem significantly different, then you test if the observed difference is significantly larger or not. This wikipedia may help you a bit further, incl a simple example: https://en.wikipedia.org/wiki/Equivalence_test

Bingo! Thank you very much to all who have contributed answers here. It was Jos' answer that pointed me to the article that I was looking for (Limentani Giselle B, Ringo MC, Ye F, Bergquist ML, McSorley EP (2005) Beyond the t-test: Statistical equivalence testing. Anal Chem 77 (11):221a-226a). One of the main reasons I find this question so interesting is that I am trying to find a rational approach to finding a robust normalisation standard in proteomics experiments. I do hope that this is pointing me in the right direction.