I would suggest re-designing the primers. Are the current ones completely complementary? I like to design the mutagenic primers in such a way that the mutations are "pushed" toward the 5'-end of the primer, and so that the overlap between them is ~20 nt in length. Let's say you sequence is ...acgcgcgcgcgaacctattccttNNactgttattaagctactttctggacgacg... where the NNs stand for your two adjacent nucleotides you want to mutagenize. Then the forward primer may be acctattccttNNactgttattaagctactttc and reverse taacagtNNaaggaataggttcgcgc.

Good luck!

It might be worth trying touchdown pcr starting from a much higher temperature than 5c above Tm. Possibly also enriching the strands separately by cycling each primer separately for 20 cycles then mixing the 2 reactions so now both primers are present and running as a pcr but with more target now competing for the primers

Its not clear for me how are you introducing the mutations- quick change, PCR, Kunkel or other method? And I cannot fully understand the kind of primer repeats you are getting. An example and the description of the method would help. But I agree that the easiest solution should be primer re-design.

Avoiding total overlap if you are using sense and antisense mutagenic oligonucleotides (as in quick change mutagenesis) is a good suggestion (see the first answer).

Another possibility is extending the primer lenght so the mutation is flanked by more nucleotides. That would change the annealing position and could result in a better annealing specificity.

Knowing more details would be required for a more precise advice.