What is you downstream application? If you need o clone the product you can cut it out from gel (assuming there is a sufficient size difference between the desired product and the others).
Their are 'n' number of probabilities for multiple banding pattern in PCR product.
If you are looking for the single band i.e. specific amplicon, then something is wrong in your protocol. I will suggest to recheck your chemicals, primers, always keep negative control to rule out contamination possibility.
I had this problem before. I guess, in addition primer concentration, annealing tempreture, etc, its better to check your Agarose gel (Percentage, Contamination,...) and Power instrument.
This could be possible chance of contamination, improper primer design (Forming dimers) or technical error. The best to rectify use negative control, check once again your primer design and use 10um for PCR. If still you see unspecific band try HOT START TECHNIQUE, prepare your PCR reaction on ICE and place on pre-heated PCR machince.
I want to take the target gene about 1-2Kb from the vector. There are many nonspecfic bands in the electrophresis. I have some adjustment on primer concentration, template contration. It is very surprised that I found many times in my lab, enve if small fragment will be amplied.