I am trying to estimate the sample size necessary for the RNA-seq study. There's no previous studies that I can refer to for an estimation. In this case, how should I approach planning for sample collection? Thank you for your help.
This depends on the sample details, experimental details but also on the type of analysis you will run on the data. Are you targeting cultured cells, primary mouse cells or primary human cells? How consistent are the sample groups in terms of cell isolation, processing, viability? How many cells are you targeting? For example, in a highly controlled in vitro experiment with >10k cells sequenced per sample, small number of discrete cell types and robust treatment effect, 2 biological replicates per group might be sufficient to analyse how the cluster composition changes with treatment and for a basic differential expression analysis. In contrast, for a study with primary patient cells in 2 or more treatment groups, even 10 patients per group might not necessarily be enough if there are batch effects, variable viability, diverse individuals etc.
You say there are no prior studies to refer to, one option is to check a few examples of similar studies (e.g. same downstream analysis, different tissue) and determine if these have enough power for your planned analysis. Another option is to run a pilot experiment which will give you some key parameters on variability, then design a follow-up experiment with sufficient power.