Using 3-AT essentially lowers the threshold for a visible result. Some bait constructs produce extensive background growth independent of the prey used, to the point that an interacting pair may be impossible to detect. Adding even 1 mM of 3-AT can help to reduce the background growth. Ideally, for any two interactors there is a concentration of 3-AT at which background growth is eliminated but growth corresponding to the interactors is still visible.
Our lab usually uses between 1 mM and 50 mM of 3-AT. Any more than that and all HIS3-dependent growth is abolished. Any less than that and we tend to see far too many false positives and positives resembling background growth.
As 3AT is a competitive inhibitor of HIS3 gene product used as a reporter gene in Y2H, the cell will be able to grow in presence of 3AT only if the level of HIS3 gene product is sufficient to sustain the inhibitory effect of 3AT to produce enough histidine to allow cell survival. Therefore the presence of 3AT will select clones with high level of HIS3 which depends on the strength of the interaction between bait and prey. The more you increase 3AT dose the less clones you get (and the stronger is the interaction).
3AT will slow down cell growth as it will slow down the synthesis rate of histidine necessary for protein synthesis. What you can do is to select clones on low 3AT concentration and then transfer transformants on plates containing increased 3AT concentrations in order to sort transformants according to the strength of interaction.