You might have a contaminated stock of your empty destination vector. If the empty vector is supposed to have the ccdB gene, then it should be lethal to the TOP10 cells. Since you are getting a lot of colonies anyway on the ampicillin plates, you could sequence some of those to see if they have a different insert. If so, then transform the empty vector again and plate on a different antibiotic (whichever is included with the ccdB gene), isolate single colonies, sequence again to make sure they are really empty vectors, and trash all the previous contaminated stocks.
If your stock is not contaminated, compare how many colonies you have between a true insert and the empty vector. If you have a lot more of the real inserts, then you should be fine (you may just have to check more colonies).
You might have a contaminated stock of your empty destination vector. If the empty vector is supposed to have the ccdB gene, then it should be lethal to the TOP10 cells. Since you are getting a lot of colonies anyway on the ampicillin plates, you could sequence some of those to see if they have a different insert. If so, then transform the empty vector again and plate on a different antibiotic (whichever is included with the ccdB gene), isolate single colonies, sequence again to make sure they are really empty vectors, and trash all the previous contaminated stocks.
If your stock is not contaminated, compare how many colonies you have between a true insert and the empty vector. If you have a lot more of the real inserts, then you should be fine (you may just have to check more colonies).
I recommend you before LR reaction, perform an isolation of your entry vector with your DNA fragment, if your entry vector has a couple of restriction sites flanking your insert try an enzymatic digestion or carry out a PCR reaction to confirm your clonation in the entry vector.
After this confirmation perform your LR reaction with your destination vector (pBABE), and transform your TOP10 cells; as control I suggest to plaque TOP10 without any vector, and TOP10 with an empty vector such as pBluescript KS or another like that.
First grow pBABE destination vector with Chloramphenicol antibiotic to check if the ccdB/Chlr cassette is intact or not. Then isolate the plasmid and transform it in your normal E.coli / TOPO cells to confirm the your vector is ok or cells are contaminated. If you don't get any colony that means the vector and cells are ok.