There may be many reasons but my opinion is that one of the main reasons is atomic radius of the doping material, which is higher or lesser than main material means it is opposite nature in band gap
Well the bandgap is the result of the electron wavefunctions that occur in a perfect lattice of a material. Once this lattice is disturbed by a significant number of dopant atoms, the allowed wavefunctions change, and thus also the bandgap.