Dear Alena,

Fisher's Exact test doesn't require anything beyond.a minimal sample size. So the representativeness of your sample will be based purely upon the bias in sampling and the size of the confidence interval around your answer you wish to have. Practically speaking, you will find that power will increase with the sqrt (n) for most questions of this type, so work from there.

Hope this helps

Dear Bruce, thanks very much for taking your time answering my question, I really appreciate it. I am new to statistics. All my data is categorical, so I think the concept of confidence interval is irrelevant (as this is directly related to mean and standard deviation). Please correct me if I am wrong.

I am really confused how to calculate a representative sample for my study? All data is categorical, unit of analysis is crypto-ransomware attack and I am examining associations between categorical variables (e.g. attack impact vs. size of organisation etc.). I have collected data on 38 attacks via structured interviews. I know 38 is not enough but what is?

Dear Alena,

I will address the bit about confidence intervals first. If you are testing the strength of a potential association between two variables you will generate a statistic. In this case, the size of the association. Fisher's test, strictly speaking, is an estimate of the probability of the given observed association under the assumption that no association exists. However, the behind this sort of statistic is that your sample represents a wider population. Accordingly we can derive an interval around "the answer" to reflect the uncertainty that the association viewed in our sample will reflect that present in the population.

These confidence intervals become smaller as your sample size grows. So your conclusion is always related to the certainty derived from the statistic observed AND the size of the sample you have. Often you only see point estimates (single numbers) representing association strengths and associated p values, but in fact confidence is always bounded in this way.

So n = 38 is large enough to investigate an association; in fact Fisher's test is generally considered appropriate as long as the smallest cell in your design has as expected value of 5+ observations under the null hypothesis.

The beauty of statistics is that if your sample is randomly sampled from the population then you can and should expect your result to reflect that population. The unfortunate part is that your ability to be certain about your result depends on the size of the association expected and the number of observations.

So to calculate a sample size you need to know two things - what are you looking for - and how much confidence do you wish to have in your result. G power and many web resources can help you explore this question and show you the relationship between the sample size and confidence. In the case of your sample I can assure you that you will not see much change in confidence until n > 60 and so on.

However, to be more helpful - how do you know that n = 38 is not enough? One answer is non-statistical - it is just that given the large number of attacks out there and the difficulties in finding people to talk to you are not confident that your sample represents the full population. In this sense you need to continue until logically you have a good representation of the population of attacks.

Most research is a compromise between the cost of obtaining the data and the precision with which the answer needs to be known. At n = 38 you already have an answer. The question is then how much better (more precise and trustworthy) can your answer become with more effort. My bet will be that you are already at the point of diminishing returns given the costs of interviewing people; unless they are saying things that cause you to question your sample on logical grounds.

Take care

Dear Bruce!

Thank you very much for your detailed answer. Although I am sure it all sounds very logical to you, I have now even more questions.

1. Do you mean I can derive confidence interval from p-value? And if yes, then how? Maybe you could point me out to some links please. Or maybe I can run it in SPSS...

2. I thought that Fisher's Exact is suitable to use when expected value is less than 5 (which is the case for some of cells in my data) (https://www.theanalysisfactor.com/observed-values-less-than-5-in-a-chi-square-test-no-biggie/)

3. My sample is randomly sampled but due to access issues to such sensitive data, there was an element of convenience. Also I find it hard to understand the concept of random sampling in the unknown population, which is again the case with my study (the population is unknown)

4. Can you please send me links of calculating sample size based on "what I am looking for" and "how much confidence I want to have in my result"

5. In terms of continuing with data collection, I am afraid this is the end for me. I might be able to collect data on a few more cases, which you said won't make difference anyway. But generally, I believe I have used all my possibilities at this point.

I am really sorry for so many questions! And I would really appreciate your reply!

With best wishes,

Lena