To determine outliers, you need to know (or at least put forward a hypothesis on) an underlying distribution of the sample which you suspect to have outliers in it. The Q-test supposes a normal distribution for the input data. If the normal model is what suits you, then it sounds safe to employ the Q-test. I by no means an expert in biology, however the description you provided allows to suggest the normal model as a reasonable one.

In any case, you may make use of the Q-Q plots (for example drawn for the case of the nomal distribution) and add confidence bands to the plot. These confidence bands will show you the outliers. If you are looking for a less conservative filter, use the Vysochanskij–Petunin inequality (https://en.wikipedia.org/wiki/Vysochanskij%E2%80%93Petunin_inequality)

To make the idea easier to understand, here is a pice of R code, which implements the above technique for getting rid of outliers (should you need more info, contact me)

# to cope with outliers we employ the Mosteller technique,

# in particular the fact that order statistics are asymptotically

# normal with the mean q_i and variance V_i, where q_i is for the

# theoretical quantile corresponding to the probability p_i=( i-1/2 )/n,

# while V=p_i*( 1-p_i )/n/f( q_i )^2, in turn f(x) is the PDF of the true

# distribution. That is, for every order statistic x_(i) the following is true

#

# q_i-u_alpha*sqrt( V_i )

Tukey (IQR)

Jochen Wilhelm Your criticism is valid: Tukey’s IQR method is not a tool for deciding whether values should be removed, but rather a graphical approach to highlight points that are distant from the bulk of the data. These values may be natural or may indicate a measurement error, and the decision cannot rely on IQR alone. Moreover, in large samples or skewed distributions, many points may appear as outliers without actually being problematic. Therefore, it is preferable to combine Tukey’s IQR with other statistical methods (such as Z-score or MAD) and with biological understanding of the data before making any decision about removal.