Fisher exact test is valid for hypergeometric sampling.
This means in a table column and row totals are fixed by the design of study.
This is a rare situation and often the sampling is different. So Fisher exact's test does not hold.
You can have a look at different study designs named Cross Sectional and Case Control whose sampling methods are not hypergeometric sampling to understand the idea.
Hypergeometric sampling is rare, there is a famous Lady tea tasting example.