It is not just the polarity that matters. It is the effect of phosphorylation of specific residues on the conformation of the protein, and also on protein-protein interactions.
Phosphorylation of any amino acid- Thr, Ser, or Tyr - can have different biological effects, depending on the specific residue, protein, and context. As you mentioned as an example, growth factor receptors, such as the insulin receptor, autophosphorylate on tyrosine residues upon binding of growth factors. The phosphorylated receptor can then recruit proteins with SH2 domains, which bind specifically to pTyr. In this fashion, phosphorylation "activates" a protein, in that it allows the protein to transmit a signal (growth factor) to downstream signaling components within the cell. It's also conceivable that a phosphorylation event could inactivate a protein - if, for instance, the phosphorylation event caused an inhibitory confirmational change, interfered with substrate binding, or disrupted a critical protein-protein interaction. So, regardless of the amino acid, most (if not all) phosphorylation events in a cell are protein-specific and context dependent, and usually require significant follow up to determine their effects on protein and cellular function (i.e., whether they activate or inhibit a protein).