There are many tools to evaluate genome annotation quality, just two of them:

Liu Z, Ma H, Goryanin I. A semi-automated genome annotation comparison and integration scheme. BMC Bioinformatics. 2013 Jun 1;14:172. 

Yang Y, Gilbert D, Kim S. Annotation confidence score for genome annotation: a

genome comparison approach. Bioinformatics. 2010 Jan 1;26(1):22-9.

A million $ question. Gene annotation is an on-going process, resource intensive, and labour demanding project. You can do whatever you have with you to play with the dataset to improve existing annotation. This is always welcome. The way how you ensure what you did is correct is a challenge. Open the data in a form of a database for community members to use. Ask for comments, reviews and opinion on the dataset from working groups. This is how we ensure quality of annotation. Hope it helps to expand your imagination. 

Maybe the date of first publication of the genome sequence can be a reasonable proxy for annotation quality -the first genomes to be sequenced were expensive, so they were probably valuable scientifically, so they probably received more attention in the annotation.

There are many tools to evaluate genome annotation quality, just two of them:

Liu Z, Ma H, Goryanin I. A semi-automated genome annotation comparison and integration scheme. BMC Bioinformatics. 2013 Jun 1;14:172. 

Yang Y, Gilbert D, Kim S. Annotation confidence score for genome annotation: a

genome comparison approach. Bioinformatics. 2010 Jan 1;26(1):22-9.

Some databanks describe the process they use to annotate genomes. For example, RefSeq databanks from NCBI uses the following key words : VALIDATED or REVIEWED for entries with some manual curation (see http://www.ncbi.nlm.nih.gov/refseq/about/ for more details). This appears in the "comment" lines of the entries.

Edit distances have been used for annotation comparisons for systematic evaluation. 

http://www.biomedcentral.com/1471-2105/10/67

In other approach, I usually take set of know gene sequences and map these on genome using GMAP and compare the overlap with multiple annotations using bed tools.

You can select the bacteria with complete genomes (not draft genomes) from NCBI. Generally bacterial genomes are easy to annotate and complete genomes are with good annotation.  Another way is to download the genome sequence of these bacteria and find genes using one or more common gene finding tool and compare the results. Manually doing the work by yourself  will be best for the type of work you have planned

When an erroneous annotation gets into a database, it can propagate. While developing a pipeline for auto-curating sequences of the nifH gene, I found a number of GenBank records that are annotated as nifH, but when RPS-BLASTed against the conserved domain db their best hits were not to the nifH domain. I noticed that these (probably) misannotated records were often submitted in batches of 10-20; my guess is that the sequences were annotated on the basis of similarity to other (probably) misannotated sequences already in GenBank, and so the error spread to the new sequences. Genomic identity theft.

Without resorting to wet lab techniques, the best you can do is drive down the probability that an annotation is wrong. This article describes how we did it for nifH; the technique seems to be applicable to other genes: Heller et. al. "ARBitrator: a software pipeline for on-demand retrieval of auto-curated nifH sequences from GenBank", Bioinformatics 2014

Also see H. James Tripp et. al. "Misannotations of rRNA can now generate 90% false positive protein matches in metatranscriptomic studies", Nucleic Acids Research, 2011. It won't help you validate the quality of annotations, but it will alert you to the extent of the problem.

HTH,

-- Phil

Maker2 have the Annotation Edit Distance for quality control. 

http://www.biomedcentral.com/1471-2105/12/491

Hi,

I would like to propose a variant of the Jonathan David Moore's answer. My proposal also uses the first genomes deposited in the NCBI. The main idea is to compare a new genome against a curated one. There are some tools where you can define, for instance, how many of similarity is expected from the new genome compared to the trusted genome. This similarity is expressed regarding size and identity (both) of genes. It's a quite simple idea but powerful because some tools can classify the genes according to this metric: genes annotated in green color has more than 95% of similarity, those annotated in yellow has between 94 and the cutoff selected by the user and finally those below the cutoff has red color and no gene name or function associated. So you can explore that with simple reports from the Artemis program: how many genes are green, yellow and red? How many genes were predicted in the new genome compared to the curated one? For this last question, the answer can be interesting; you can figure out that your genome can be full of frame-shifts because the number of genes colored in red is larger than or close to the number of genes in the curated genome, so bigger than the expected for the new genome! Such tools also use to deliver a report indicating which genes could be frame-shifted compared to the curated genome.

For a while, please, use the below temporary site because the official website is under maintenance to substitute the old version for a new one. The current version supports a multifasta file to be annotated, and each fasta will be separately annotated.

BTW, in the below site just ignore the re-captcha resource.

Regards,

Anderson Santos

Citation:

Santos AR, Barbosa E, Fiaux K, Zurita-Turk M, Chaitankar V, Kamapantula B, Abdelzaher A, Ghosh P, Tiwari S, Barve N, Jain N, Barh D, Silva A, Miyoshi A, Azevedo V. PANNOTATOR: an automated tool for annotation of pan-genomes. Genet Mol Res. 2013; 12(3):2982-9. [PubMed: 24065654]

http://pannotator.facom.ufu.br