Following the logic of the whole planning procedure this means that you are better off spending your time and money in a different project or question.

As Peter said it is not prohibited to do the study anyway, hoping for the best. But resoning will be kind of stupid, as it will be in the line of "we thought a lot about planning the study, and after a careful consideration of all available information we found that we would need to analyze at least N subjects to have an acceptable expected loss (or win) with the study. Now we have done the study with only N/2 subjects - who cares about the expected loss/win ?!"

But maybe you should tackle the whole problem starting from the available sample size. Given this size, what power will you have for the minimum relevant effect size, or what effect size could you detect with an acceptable power? If the effect size is large enough to justify a respective action or if the achievable power is high enough to justify the expected loss (or win), then the sample size is justified and the agrumentation makes sense. If neither is the case, then conducting the study is actually not justified.

It is kind of playing lotto. You may play it for fun, hoping to win something. However, if you really take care about the money you spend and the money you may win, the expected loss becomes relevant. And as soon as you find that playing lotto has an inacceptably high expected loss it would simply seem stupid to play lotto anyway ("lotto is a tax for people who are bad in probability theory" - not my own quote, the author is unknown to me).

Dear Hendrik,

There is nothing stopping you running your experiment with a smaller sample size. It just means that your experiment will not be as sensitive at detecting an outcome, such as a correlation or a difference between groups. You should then mention this in your discussion.

Following the logic of the whole planning procedure this means that you are better off spending your time and money in a different project or question.

As Peter said it is not prohibited to do the study anyway, hoping for the best. But resoning will be kind of stupid, as it will be in the line of "we thought a lot about planning the study, and after a careful consideration of all available information we found that we would need to analyze at least N subjects to have an acceptable expected loss (or win) with the study. Now we have done the study with only N/2 subjects - who cares about the expected loss/win ?!"

But maybe you should tackle the whole problem starting from the available sample size. Given this size, what power will you have for the minimum relevant effect size, or what effect size could you detect with an acceptable power? If the effect size is large enough to justify a respective action or if the achievable power is high enough to justify the expected loss (or win), then the sample size is justified and the agrumentation makes sense. If neither is the case, then conducting the study is actually not justified.

It is kind of playing lotto. You may play it for fun, hoping to win something. However, if you really take care about the money you spend and the money you may win, the expected loss becomes relevant. And as soon as you find that playing lotto has an inacceptably high expected loss it would simply seem stupid to play lotto anyway ("lotto is a tax for people who are bad in probability theory" - not my own quote, the author is unknown to me).

I agree totally with Jochen. This is a textbook reason why power analyses are recommended: to STOP people doing under-powered studies. Besides the them more likely being a waste of time (if one is just interested in hypothesis testing) as Jochen says, they will also have high likelihood of what Andrew Gelman calls a Type S error, where you find a significant result in the wrong direction. It also has a really high likelihood that if you do find a significant effect that you have made a Type M (Gelman's terminology) error, over-estimating the magnitude of the effect. So I am more pessimistic than Jochen, even if it appears that you mean the lotto with a low powered study being significant, the result may be an M or an S error, 

Finding low power is a good reason to change the design. This might mean using more reliable instruments, doing longitudinal designs, etc. It will depend what phenomenon you want to research. There should be several ways to research the phenomenon.

I think Daniel's valid points are difficult to understand, because Gelman's S- and M-errors do not apply for a A/B testing but fo significance testing. There they are relevant in the "bigger picture" of our scientific system, where moslty only the "significant findings" will be reported, leading to a publication bias.

Small studies can be conducted very often, by many groups, and the entire study may be repeated if the result of the first was not satisfactory or publishable. The problem here is that the interpretation of the significance (p-value) depends on the entire context, including the number of studies made on the subject/topic, and that this context is not reported. The lottery example is still good here: Under the hypothesis that humans only guess the numbers, winning lottery is very unlikely. So the event of winning it is highly significant, implying that guessing is rejected as a possiblity (in favor of the existence of psychic powers, for instance). However, people do win the lottery, and yet guessing is remains a good explanation. Because we know that many many people play it very often, and sometimes even unliekly things happen. Considering the total number of "experiments" that were performed to win the lottery, the low p-values of a single winning of a particular person is not regarded as "significant" evidence against the hypothesis that humans are guessing. But in science, only this (or the few cases) would be published, stating a tiny p-value and claiming a "significant result" that would not be significant in light of the whole context. Statistically, reported results will be biased towards too-large effect sizes and too-low p-values.

However, this is a bit different for A/B testing (the formalism proposed by Neyman and Pearson; I used to call it "hypothesis testing" because two substantially different hypotheses A and B are tested, and the test should give a decision criterion telling us if it would be better to act as if A was true or to act as if B was true. This is very different in Fisher's "significance testing", where only the significance the available data under a unique "null hypothesis" (without alternative!) is assesed in form of a p-value, and where this null is rejected when the significance is sufficiently high [the p-value is sufficiently low]. Strictly speaking, neither the concept of "power" nor the concept of "confidence" do apply here, at least not in the sense intruduced by Neyman/Pearson. The A/B-testing is not about rejecting a null but to decide between A and B. These two philosophically very different approaches are unfortunately mixed and confused in many textbooks, leading to a lot of fluster in statistics.

The formal power analysis is part of A/B testing. Usually A is the hypothesis of "zero effect", and B is the hypothesis of the "minimally relevant effect". power and size of the test (i.e. the values of beta and alpha, respectively) are chosen based on some loss function accosiated with scenarios in one of the hypotheses would actually turn out to be a good description of the world and when we act according to this hypothesis or according to the other hypothesis. The result should not be reported as "significant". The result is to decide for B with confidence 1-alpha or to decide for A with confidence 1-beta. The confidence is given by design. The actual p-value that might be calculated from the observed data iis meaningless in this context (the only connection is that one would decide for B when palpha, but that decision can be made based on defined acceptance regions of the test statistic, so that the explicit calculation of a p-value is not even required).

Hence, if the design of an A/B study makes sense, then the result of the study is sensibly interpretable - in any case. One does get the data to decide if one is better off behaving as if A was true or if B was true with the desired confidence. Note that defining the design, especially chosing useful confidence values, is based on the whole context of the study. It makes a difference already here if I know that the study will be one of several similar studies. Actually, the logic of this procedure would break down in such a case (doing several well-designed studies on the same question is surely a waste of resources - just what should be avoided by desining the studies well!).

That's almost opposite for significance testing. Here, whatever data is available is used to calculate the significanc of the (available) data under some null hypothesis. A low p-value indicates that there is simply sufficient data to recognize that the model restricted to the null hypothesis is clearly enough unable to describe the properties of the observed data. Therefore I  see the p-value or the significance as a proxy to asses a "statistical signal-to-noise ratio". If this is good enough, then I may start interpreting at least the direction of the effect. Otherwise the signal is too low and/or the nois is too large to allow me even to interpret the direction of the effect, so I consider my data as inconclusive or insufficient. So, having a "significant result" means: "I do have enough data to interpret the direction of the effect". Not more and not less. To say this again: the interpretation of an actual p-value depends on the whole context. Although it is unfortunately quite common practice, it is not very smart to use a fixed threshold (like 0.05) irrespective of the context.

The ideal situation is have a enough sample, if it's very expensive, you could work with anothers groups with economics resources, then you will make the experiment with a good sample. But, it's not possible, you could use some tool for to expand the sample, tools lika a jacknife or monte carlo simulation, the sample will more bigger and will keep the same behavior of the original sample.

can i ask what is your N and what H0 you want to test?

Thanks everyone for the answers so far.

H0 is that a treatment has no effect on the outcome variable.

Say I calculated an N of 230 (115 per group) in order to detect an effect size of d = 0.44 with a power of roughly 80%, using a Tobit model with outcomes left censored at 0, and alpha = 0.05. Now I only manage to gather 120 (60 per group) observations, which would increase the minimum detectable effect size with roughly 80% power to d = 0.58.

What could be my line of argument in order to defend the statistical finding (either able to rejet H0 or not) I get with the 120 observations?

It means that you will have a higher probability of type II error, i.e., it will be more likely that you will not be able to present as significant relationships that ARE significant in the population. By "higher" I mean higher than recommended. I think that still it is better to run power analysis and report that you were not able to collect the recommended sample rather than reporting no power analysis as many researchers do.