This question is more difficult to answer than you might think.

Actually, a power analysis (=determine a sample size required to have a desired minimum probability of rejecting a given H0 with a given confidence) requires to have a "good" idea about almost everything (that the model used is appropriate, that the variables behave as they are supposed to, that everything relevant is considered, and how large the "true" effect and the residual variance are. Almost nothing of all this is actually known (if it was known, we would'n do an exeriment!). So this is all based on assumptions. We assume having a good model, we assume observing all relevant variables, etc. The same we must make resonable assumptions about the effect and the residual variance. This can be done by conclusion by analogy and is largely influenced by expert knowlede of the subject matter.

For sure, pilot data may help, but it remains very difficult even with pilot data to come up with a resonable guess. Pilot data means small sample size, and these small samples are supposed to be used to estimate an effect size and the residual variance. But estimation is a much more tricky problem than testing. You would need way more information (=data) to get an estimate that is reasonably precise to be confidently used as a guess for the power analysis... Just using a point-estimate (obtained from some small sample from "pilot data") may be way off and far from warranting you the desired power. The main problem is to estimate the residual variance, as it is notorously difficult to estimate a variance, and the distribution is not symmetric (it's Chi², a heavily right-skewed distribution).

I agree that pilot data can give you some impression or idea, but eventually, "just taking the point estimates" for the power analysis is usually not a good idea. You must reflect some expert knowledge and possible "adjust" these estimates.

But whatever you do, at the end you simply don't know what actual power you will have, but at least your plan is reasonable.

Having said all that, I think it's reasonable to use some values from the study to get an idea how large the sample size must finally be (roughly). The values you eventually plug in for the power analysis should anyway not be based only on the point estimates you get from this "pilot data", and after all, these values you use are your assumptions. It does not really matter how you came to these assumptions, as long as you can understandably argue why you think that they are reasonable (given all things known in your subject area).

Given the fact that the work already started and that the data will be analyzed one way or the other, another option is to allocate the maximum sample size that is practically feasable (given that the student has to do other projects as well and has a limited maximum time to spend for all projects of his thesis) and then just see if the test of the relevant H0 is significant. If this is the case, then the data can be interpreted w.r.t. to H0, and otherwise it remains inconclusive. This would be unfortunate, though, but it would not have been possible to get more data anyway, so this was already the best you could do.

Thank you very much Jochen Wilhelm for this detailed and quick answer, this is super helpful!

On the p-hacking issue, I assume you mean the data will not be used, not the stats. Obviously looking at the data, not the particular stats you estimate, is the issue.

Mumtaz Ali Memon, "post-hoc power analysis" is really something different. Despite the fact that post-hoc power analysis is utter nonsense, it is supposed to serve a completely different purpose.

You write in your own linked paper that this is not about post-hoc power analysis (1st paragraph on page x), and the paper also adresses not the problem Mélissa asked. So I wonder what the motivation of your answer is. Are you trying to advertise your paper?

Daniel Wright thank you for your answer,

What I meant is the following:

The reason why we want to run the power analysis is not because we do not have satisfactory results with the current sample size: in fact, the statistical tests that we conducted so far are not good and we won't keep them for the final paper, the p-values that we have at the moment tell us nothing.

I think that the point you raised is actually my question:

Let's say that I have 100 data points at the moment, and I want to know if I need more to have a reliable study [I know that we should have done that before, with pilot data, but we did not]. I run a power analysis on these 100 datapoints, and it tells me that I need a final sample size 150 datapoints. My question is: Can I include the 100 data points in the final analysis, or shall I collect 150 new data points (and dump the current 100 data points)?

I hope that I clarified your doubt,

Sorry for taking this long to answer, Research Gate suddenly decided not to notify me with your answers!

Hi Mélissa Berthet , what you would need to do is set up a stopping rule, for example, power analysis is usually used to say what sample size to stop at, but you could use things like waiting until the standard error is, say, equal to 2 centimetres. You can't use something in the hypothesis, so this typically means not looking at the means or related statistics. It is tricky not to look at anything other the specific criterion for stopping, so for example looking at most plots would provide you with information about the hypothesis. This is one of the reasons some people argue for Bayesian methods..