It's true that nonspecific interactions are a factor, especially as baits and preys are usually expressed at levels higher than in their usual conditions and in cellular locations where they may not usually be present. That's not even considering differences in protein background: a bait and prey pair from any bacterial or plant species is in an entirely different cellular background in yeast than among its usual cellular proteins.
I would argue, however, that the false positive rate seen for Y2H is due to methodological reasons as much as mechanistic ones. Filtering out "sticky" proteins and those likely to produce far too many interactions to be biologically relevant (e.g., chaperones) improves results and reduces false positives, as does performing replicates under alternative conditions (e.g., different N- and C-terminal fusion proteins for the baits and preys, different concentrations of 3-AT, or even just different protein expression conditions or interaction screening methods). Interpreting Y2H interactions can also be difficult without some computational work and filtering so I suspect that adds to the apparent false positive rate.