The most fundamental pillar of QSAR is the presence of a well defined chemical series in which all the molecules are 'slight variations' on a common theme (see attached paper) this implies a shared pharmacophore. This implies you can look at any chemical distance based on molecular graphs (see https://www.sciencedirect.com/topics/pharmacology-toxicology-and-pharmaceutical-science/applicability-domain) so to give a quantitative appeal to your applicability domain reliability, but the crucial point is the homogeneous series. As a matter of fact QSAR models only the 'disturbance' on a shared signal due to few sunstituent groups on the same moiety.

https://www.sciencedirect.com/topics/pharmacology-toxicology-and-pharmaceutical-science/applicability-domain

There are two general types of AD measures. Novelty detection, .i.e. the distance of the predictedobject in descriptor space (typically the descriptor used for the model), and method- inherent predictability. The second expressed by e.g. the percentage of trees voting for the predicted class in case of a random forest, the distance from model center (covariance matrix based), support vector length (SVM) etc etc, error metrics in case of regression models.

Problems are:

1) novelty does not per se limit predicability

2) the other metrics are not at all comparable between different methods

3) error metrics give a mean error for the model at the point in time it was validated but not an individual error.

There is a good paper by Knut Baumann on the first two metrics.

In an attempt to overcome some of the issues there were attempts to e.g. learn a model to predict a model, create multiple metrics and combine them, bin the error expections for the different values ranges of the dataset (there are typically more data points around the mean than the extremes. See papers by Sheridan from Merck. There are also OECD guidelines about best practices.

The only way to get a feeling if it works is to look into the classification/prediction accuracy if you remove predictions with low confidence score and look, if the model gets more predictive.

But your users will dislike a good model that does not predict half of the molecules. There is always a trade off.