Indeed it is possible that the polymerase is chewing back your oligos, therefore be sure to assemble the reaction on ice and add the polymerase at the very end. Are you using the suggested ammount of dNTPs? You can also try a hot start, adding the enzyme after denaturation.

I hope this is helpful, let me know.

Best,

E