I have applied colony PCR to differentiate bacterial and yeast isolates (recently published work) and my experience with colony PCR is very positive. My suggestion is to use a ready-made PCR mix to minimize a possibility for mistake and to just touch the colony (as Katie recommended). Both too much and too little of the template DNA would lead to the negative or poor outcome, otherwise the results are quite consistent and reliable.
I think the main disadvantage is the variability in the assay. Depending on how much bacteria you add you may inhibit the PCR (there are variation were you use a lysis buffer before the colony PCR that may alleviate this a little bit). Also since you are using this for bacterial identification different bacteria will have different cell wall compositions therefore require a different strategy for lysis prior to PCR (for example lysozyme for gram positives). In short, your main limitation is creating a protocol that can reliably extract DNA that is clean enough for PCR.
I have used the Colny- PCR to confirm a deletion/existence of some gens in E. coli strains and Salmonella and it works very well using ReaddMix (PCR Master Mix).
In colony PCR, a colony growing on solid media is used as a template in PCR. One of the main challenges with colony PCR is the possibility of false negative results in the PCR as there are chances that template is not well lysed for PCR. I advise increasing the time used for the initial denaturation of the template in the PCR in order to reduce on false negative results.
Colony PCR is not done to identify bacteria, but it is a short cut way to check if the transformation was successful or not. After getting this answer, one might decide a strategy to proceed further.
I would want to add that colony PCR could be problematic if the region on your template -region your using for your bacterial identification is of longer kb - , secondly in colony pcr the DNA prep may come with impurities though that may be reduced or minimized by a dilution series
It's not the most reliable method. As other's have said, you run the risk of false positives (from incorporated plasmid) or false negatives (inhibited PCR). I know folks who use it as a "first look" for a successful transformation, but always recheck by growing an overnight liquid culture and doing a plasmid prep & PCR and/or restriction digest.
Using fewer cells seems to make the protocol more reliable. You don't want to use a visible "blob" of cells, just barely touch the colony with a probe or tip and then stir into your PCR mix.