There is no "formula". Removing outliers is a judgement call. If you do remove an outlier, you must be honest that you did and be able to provide good justification for why.
you'll need metrics to define what an outlier is. best ones are box plots or calculate a PCA (Principal Component Analysis) that should regroup your samples by groups. a real outlier won't be in the good group. but you'll also need to investigate why your sample is an outlier (quality of sample? problem during sample preparation? real phenotype?....) and verify if you can discard it (enough number of samples?).