a) is correct, b) is not.

That is just the key issue of frequentist statistics that an individual p-value or an individual conficence interval (CI) is meaningless. It is only the benchmark used to control error-rates in (hypothetical and infinite) repetitions of "similar" experiments. P-values and CIs are NOT interpreted with respect to a single outcome. They do not show you how reliable a particular result is (how well the result represents a "true" effect).The reliability is introduced as part of the strategy (not as part of the results), that is, when you define the alternative hypotheses and the desired error-rates. The formalism of the data analysis will lead you to accept one of the two alternatives, and the confidence associated with your decision is not given by the actual outcome but by the error-rates you have chosen and after which you designed the experiment.

In your proposed statement you mix incompatible concepts. With power-analysis and CIs you are in the realm of Neyman/Pearson hypothesis tests. There is no "null hypothesis", only two distinctively different alternative hypotheses (usually one of them indicates a "zero-effect" and is often confused with the null hypothesis, and the other is a specific non-zero value [and not just "anything else but zero"!]). The alternatives are chosen in relation to the problem, and it would not make sense to allow a reader to interpret this data differently (because this data was generated in an experiment that was specifically desined to test these two alternatives, with the given model, and with the given confidence). Just chosing a different confidence (or error-rates) after the experiment would render the experimental design useless; it would be an out-of-context interpretation of the data. If an interpretation of the results in a different context is wanted, the entire power-thing and hypothesis-testing is quite meaningless.

In contrast, Fishers significance testing does not know the concept of error-rates, confidence intervals or power. Here, the compliance of some observed data with a given model is analyzed. The model should be rich enough to give a good description of the data, but for the test it is restricted in a parameter which is called the null hypothesis. A miserable compliance (measured as a "p-value") under this restriction is used as indication that the restriction of the model (i.e. the null hypothesis) is to be rejected. A compliance may not simply be interpreted as indicative for the restriction, because both, model and restriction are theoretical concepts that will be logically false anyway. The aim is only to demonstrate that the available data is already sufficient to show this.

Neither of these two philosophies, the Neymanian hypothesis testing and the Fisherian significance testing, does address the question that is asked by the researcher (and this is what is so terribly confusing!). What we actually want is a statement about how much the data makes us believe in a particular model (more specifically: in a particular value of a parameter or effect). This is a Bayesian approach. So the price we would have to pay to get answers to our questions is to accept a subjective view of probability. This leads us to two possible solutions: either we want to present "objective" conclusions, but then these concusions are about questions we did not ask - or we answer the questions we actually have, what is possible only on a "subjective" basis.

But to come back to your statement: you can report confidence intervals, but they can only be interpreted as a "statistical signal-to-noise-ratio" (just like p-values, that are the same thing, viewd from a different angle). There is per se no information about a reliability of these particular results or conclusions. It is simply not there, no matter how much we wish it should be.

Your proposal sounds good to me. Indeed, post-hoc power analysis is not useful, so if someone wants it it's better to just put confidence intervals (ideally, confidence intervals should always be there anyway). In fact I had a paper where I was making an argument from an important nonsignificant effect; a reviewer wanted post-hoc power analysis, but I was able to get away with just adding CIs and putting a [meaningless] post-hoc power analysis in a supplementary file to keep the reviewer satistifed: http://journals.plos.org/plosone/article?id=10.1371/journal.pone.0063943

a) is correct, b) is not.

That is just the key issue of frequentist statistics that an individual p-value or an individual conficence interval (CI) is meaningless. It is only the benchmark used to control error-rates in (hypothetical and infinite) repetitions of "similar" experiments. P-values and CIs are NOT interpreted with respect to a single outcome. They do not show you how reliable a particular result is (how well the result represents a "true" effect).The reliability is introduced as part of the strategy (not as part of the results), that is, when you define the alternative hypotheses and the desired error-rates. The formalism of the data analysis will lead you to accept one of the two alternatives, and the confidence associated with your decision is not given by the actual outcome but by the error-rates you have chosen and after which you designed the experiment.

In your proposed statement you mix incompatible concepts. With power-analysis and CIs you are in the realm of Neyman/Pearson hypothesis tests. There is no "null hypothesis", only two distinctively different alternative hypotheses (usually one of them indicates a "zero-effect" and is often confused with the null hypothesis, and the other is a specific non-zero value [and not just "anything else but zero"!]). The alternatives are chosen in relation to the problem, and it would not make sense to allow a reader to interpret this data differently (because this data was generated in an experiment that was specifically desined to test these two alternatives, with the given model, and with the given confidence). Just chosing a different confidence (or error-rates) after the experiment would render the experimental design useless; it would be an out-of-context interpretation of the data. If an interpretation of the results in a different context is wanted, the entire power-thing and hypothesis-testing is quite meaningless.

In contrast, Fishers significance testing does not know the concept of error-rates, confidence intervals or power. Here, the compliance of some observed data with a given model is analyzed. The model should be rich enough to give a good description of the data, but for the test it is restricted in a parameter which is called the null hypothesis. A miserable compliance (measured as a "p-value") under this restriction is used as indication that the restriction of the model (i.e. the null hypothesis) is to be rejected. A compliance may not simply be interpreted as indicative for the restriction, because both, model and restriction are theoretical concepts that will be logically false anyway. The aim is only to demonstrate that the available data is already sufficient to show this.

Neither of these two philosophies, the Neymanian hypothesis testing and the Fisherian significance testing, does address the question that is asked by the researcher (and this is what is so terribly confusing!). What we actually want is a statement about how much the data makes us believe in a particular model (more specifically: in a particular value of a parameter or effect). This is a Bayesian approach. So the price we would have to pay to get answers to our questions is to accept a subjective view of probability. This leads us to two possible solutions: either we want to present "objective" conclusions, but then these concusions are about questions we did not ask - or we answer the questions we actually have, what is possible only on a "subjective" basis.

But to come back to your statement: you can report confidence intervals, but they can only be interpreted as a "statistical signal-to-noise-ratio" (just like p-values, that are the same thing, viewd from a different angle). There is per se no information about a reliability of these particular results or conclusions. It is simply not there, no matter how much we wish it should be.

Hendrik - 

You note "We report confidence intervals on our estimated regression coefficients...."  If you have regression, you could estimate a prediction interval about the predicted y-values.  

Below is a link to a simple example.  For multiple regression, you might let a preliminary prediction of y be the size measure, rather than x. 

https://www.researchgate.net/publication/263265199_CRE_Prediction_%27Bounds%27_and_Graphs_Example_for_Section_4_of_Properties_of_WLS_article

Test data would also be good.  The estimated variance of the prediction error does not directly cover bias, though it does to a degree, incidentally. 

Cheers - Jim 

Data CRE Prediction 'Bounds' and Graphs Example for Section 4 of ...

I should also add that, technically, "some treatment-effects are within the confidence interval and are therefore likely to represent the true treatment effect" is not strictly true because a confidence interval does not necessarily always represent plausible parameter values (see https://learnbayes.org/papers/confidenceIntervalsFallacy/)

Hendrik -

Type I and type II error probabilities together give you similar information to a confidence interval.  Both approaches are impacted by sample size, both depend upon standard deviations, but confidence intervals are generally more easily interpretable.   The connection may best be seen by considering a sequential hypothesis test.  In that case, you pick two 'simple' alternative hypotheses, and sample until you have the specified level of ... let's say ...'confidence' ... that the 'truth' is closer to one of those hypotheses than the other.  The difference between such hypotheses would indicate "effect size." 

You might be interested in this:

 https://www.researchgate.net/publication/262971440_Practical_Interpretation_of_Hypothesis_Tests_-_letter_to_the_editor_-_TAS

But, I wonder, reading and rereading your question, if you might really be considering impacts on regression.  If you are looking at the coefficients of regression, for multiple regression, and comparing them, relative to one another, then either confidence intervals, or with equal sample sizes, even p-values might be used, if you are looking at multiple linear regression (but perhaps not so meaningfully for nonlinear regression).  Except, even then, there are often interactions between these regressors which muddy the meaningfulness of such comparisons.  Also, it is the value of a coefficient times such a variable value which determines the overall contribution to a predicted y.  Other regressions may become much more complicated. 

If you are looking at regression, then you might be much more interested in validation for competing models, such as cross-validation techniques you could find in a textbook such as The Elements of Statistical Learning, Data Mining, Inference, and Prediction, 2nd ed, 2009 (corrected at 7th printing 2013), Trevor Hastie, Robert Tibshirani, and Jerome Friedman, Springer.

I earlier also noted the estimated variance of the prediction error.  The square root of which could be found for some regressions in SAS PROC REG, for example, where it is output as the STDI.  Prediction intervals, based on the square root of the estimated variance of the prediction error, just like confidence intervals, are reduced by increased sample size.  This is just as any standard error is based on a standard deviation, and reduced by increased sample size. 

It just seems that what you are probably really looking for is to compare performances of models.  It could as simple as looking to see if you do better with or without a given independent variable.  (Some might suggest "principle components," though that impairs interpretation.)  But perhaps you had something else in mind. 

If you are comparing the relative influence of regressors, they may interact enough to make that problematic.  If you are looking at sample size, type II error analyses and/or confidence intervals, all based on standard error can help, but overall, when it comes to predicted y, you would want to know about the estimated variance of the prediction error.   You ask yourself, What sample size makes the estimated variances of the prediction errors acceptable?

Cheers - Jim

Article Practical Interpretation of Hypothesis Tests - letter to the...