does not seem reasonable for me. confidence intervals (and testing problems) are not linear in terms of sample size, but alpha / p would be linear.

How do you measure p ?

It does not seem reasonable to me. The question is rather vague. True ,the sample or data contain information about the population. But what is this information and how do we measure it? Is it about parameters of the population or is it information (uncertainty) of or in X?

If we have a nominal level of significance alpha,  what does it mean to have an actual level of significane alpha/p. Do you imply that in testing problems we should use alpha/p instead of alpha?

Adnan -

I do not completely follow this, but it sounds like the more information you have, the lower you want to set a threshold alpha for 'significance' for a p-value. That would make sense to me. A p-value is a function of sample size. The larger the sample, the greater the information (presumably), so the smaller the level should be set. When people use the same level, say 0.05, as is often done, then a large enough sample size will tend to reject any point ("simple") null hypothesis, no matter how close to true it is for all practical purposes, as it is not going to be "exactly" true. Also, with a small enough sample size, the null hypothesis will not be rejected, often even when far from true. So your statement seems true enough if it means it is more reasonable to set your level according to the information (sample size) used. Defining "information" so that this works in every case would be tough though.

A power analysis should be done, not to pick a test, but to compare hypotheses, or some other sensitivity analysis done. A sequential hypothesis test might help.

This is why I prefer a confidence interval about a simple hypothesis (point estimate), when feasible, instead of hypothesis or significance testing.

Cheers - Jim

Article Practical Interpretation of Hypothesis Tests - letter to the...