Good references on sample size for quantitative study when sampling frame is big, say more than three hundred thousands and the listing of all individuals is not possible?
It isn't just how big the sample size may be that matters, it's how the sample is constructed. If you want to be able to infer from a sample to a population, there are two basic ways: (1) by a randomized selection, where the inverse of the probability of selection is used to weight the sample up to the population, or (2) by use of regression modeling to relate the sample data (y) to regressor data (x), where the x are known for the population. Or (3), a combination of (2) and (3). Otherwise you can have seriously biased results. Even (2) alone can be risky.
It is often good to stratify (categorize) your population, regardless, to reduce uncertainty.
A sample size, from your reduced-bias sample, is then determined by how large it needs to be to reduce, typically, standard error of a mean or total, and this also depends upon the details of your sample design, and population or subpopulation standard deviation(s). And this assumes you do not strain resources and lose data quality.
With a large size for each subpopulation or stratum, you may not need to worry about a finite population correction factor.
Because you cannot cleanly identify your population, I assume that presents a challenge in drawing a "representative" sample as described above. Perhaps you are in a situation where you need an area frame, where your sampling unit is at a higher level, and you use a two-stage sample where you first choose among the larger units, and then sample or census at the level you want within the larger units chosen. Therefore, you may want to research the following terms: "multistage sampling," "area sampling," and "cluster sampling."
If you are doing a nonprobability sample with no regressor data, then results can be substantially biased, and there is no way to really measure standard error and no sample size requirement 'formulas.' The best you might do is to divide your population into categories and be sure to sample from each, and repeatedly sample enough that your mean results in each category stop changing to an unacceptable degree ... meaning unacceptable to the certainty your subject matter and application requires.
At any rate, it might be best to start with a pilot study.