ISSUE: Sample size

WRONG SAMPLE SIZE: The fact that the total population of 180 million was brought up, automatically one sample calculation method call Yamane method must be dispense with as inappropriate because N here is assumed unknown. The 180 million include "irrelevant targets" as well, i.e. people in remote regions of the country that would otherwise not be part of the study. This wrong sample size method is given by:

(1)   nY = N / (1 + Ne2)

... where N = 180,000,000 (which is not reasonable for reason stated above); and e = error level, say, 5%. The result is the "magic" number of n = 400.

Who are your targets? How many of these targets are out there? This is the real N. Generally, this is non-finite.

NON-FINITE POPULATION: Assume that the population is non-finite, i.e. N = unknown. Why? Because 180,000,000 is the country's population. It is not the frame for the sampling. For non-finite population, you need to take series of test samples to learn about the characteristics of the population. Then use the non-finite population minimum sample population:

(2)   n = Z2σ2 / E2

... where Z = standard score at a given % confidence interval; σ = estimated population standard deviation; and E = standard error; generally given as E = σ / SQRT(ntest). How large should the test sample size be? Adequate size for the manifesting of central limit theorem, i.e. size = 30. Do series of test, bootstrap the tests.

ONLINE SAMPLE & FIELD SAMPLE: If you obtain 1500 from online and 500 from the field, you need to test both first whether they may be combined. The online population and the natural person that fill out the survey may not come from the same source, i.e. they come from different population thus manifesting different distribution. If one source is normally distributed and a second source is non-normally distributed, you have two very distinct population that arguably manifesting distinctive behavior---combining them would invariably create inferential error. Here Type I error if you insisting that two distinct and different populations are "the same" thus combining them in one sample.

If sample tests show that there are no significant different between the two samples: online and field, then combining then is no problem. If the test shows that they are different, you have the bifurcate the samples into 2 samples and engage comparison study. For applicable test, see attached file.

AVOID INTERVIEW: I assume this is face-to-face verbal interview. Generally the data is unreliable due to bias: judgmental error. Avoid this if you can. If you interview 23 people, you have potentially 23 different answers which you have to transcribe, and code. The information may be lost during transcription and coding. If you have expert interviews or focus group, do it for narrative report not for quantitative analysis.

you need to define your population. who are the target groups, that is your population. Ethics dictates (different for different countries/cultures) you cannot survey respondents below a certain age without special ethical approval. The sample size can be determined by formulae (Yamane's mentioned above) and no matter what the population size, with a 5% margin of error built in, the sample size does not usually exceed 400

You need to be very specific on who the respondents should be then you need to identify some demographic you know in that population relevant to the research topic e.g. gender , age groups etc.and represent the sample by the same proportion of your chosen demographic (a representative sample)

So if you do determine the size to be 400 say then for it to be representative you need the same demographic split for representativeness. 

Typically there are approximately nine different sampling methods - you can use individual ones or combinations, each has their own set of rules and representativeness can differ based on the method(s) employed.

for online versus self administered you could check reliabilities for the different groups separately to observe any possible differences - there are also some other statistical checks you can apply to observe possible differences

It appears that your biggest problem is not mixed mode, but that it does not appear that you had a sample design. Without a probabilistic sampling design (say, stratified random or cluster sampling), or perhaps auxiliary (regressor) data on the entire population for modeling in appropriate groups, or both, you cannot draw reliable inferences about the entire population. Sample size needs depend upon your plan, the kind of data, and its variability.

A randomized (probabilistic) sampling plan would have likely been best for you. Without regressor data on the entire population you likely have something far less reliable, such as a convenience sample.

Perhaps you could do some descriptive statistics on the data you have, separated into groups you hope are as homogeneous as possible. A confidence interval for a mean for a given such group, however, would also assume a random draw from the group it would hopefully represent, which may not be nearly enough true, and is definitely another source of unknown error. However, you may be able to get some idea of the differences between various groups , including the same categories with different collection modes, and perhaps follow-up with a more scientifically designed new study later.

You could take a look at some survey sampling and estimation books. The first few chapters of this may help:

Cochran, W.G.(1977), Sampling Techniques, 3rd ed., John Wiley & Sons.

Or you might try this:

Blair, E. and Blair, J.(2015), Applied Survey Sampling, Sage Publications.

If in a follow-up study you might compare data collection mode change for one set of respondents, the attached link might be helpful, depending upon how you do it. The link is included in case it gives you any ideas for future testing.

Modeling data with the model-based form of the classical ratio estimator (CRE) may be helpful. In Lohr, S.L.(2010), Sampling: Design and Analysis, 2nd ed., Brooks/Cole, chapter 4 - I think - design-based and model-based CRE is discussed in a relatively clear manner.

But I think you are going to best expend your efforts on strictly design-based (probabilistic) sampling and estimation methods, and the first few chapters of Cochran or a similar book should be very helpful.

Descriptive statistics and graphics on the data you have already collected may be helpful in learning about your data and the questions you need to answer and may serve as a kind of pilot study for a new effort.

Cheers!

PS - Sample size considerations are - for example - a chapter in Cochran(1977).

PSS - If you are using likert scales, I am not really familiar with them, but I suspect that your concerns should be similar to most of those above. Cochran considers continuous data and also proportions ("yes/no" data). 

Technical Report A Note on Estimating Impact of Data Collection Mode Change f...

Dear Paul, Robert & James,

I am so great full to you for such an extensive reply from each of you. Please give me some to go through your comments / suggestions and then I will come back to you for further guidance (if required). Thank you very much for your valuable comments and precious time, otherwise I was feeling so lost.

Regards

SAM

You can use Krecjie and Morgan sample size determination to validate you sample size. However, anything is considered infinite; per the Krecjie and Morgan formular you can select only 384 respondents which will be representative of the population.

Check: https://www.google.com.gh/search?q=krejcie+and+morgan+sampling+table+pdf&tbm=isch&source=iu&ictx=1&fir=E3503dxzJx2dzM%253A%252C34M50cTrVrF1CM%252C_&usg=__LmT0NutV-tbOSmUJeHxAIfNjRfk%3D&sa=X&ved=0ahUKEwjnrd-spfjYAhVDvxQKHXFeAjIQ9QEIOTAC#imgrc=E3503dxzJx2dzM: