What you state is not true - the identity of suitable dopants of course strongly depends on the semiconductor. Even if Si is usually doped with B, P, As, or Sb (groups III and V), it can also be doped with group VI elements such as Se (to make it n-type).
Just to give you an example: III-V compounds (such as GaAs or GaN) are an important class of semiconductors, and these quite naturally need to be doped with elements other than group III or V. Typical dopants for GaAs are Si (group IV) and Te (group VI). p-type GaN is achieved with Mg doping (group II).
Morteza Ajami The fact that we do not use group 12 or group 16 elements for doping is because of instability. Consider the situation something like this. There's a group 12 element say zinc. It is added as an impurity to the intrinsic semiconductor(say silicon, having 4 electrons in the outermost shell). What happens is that although zinc contributes 2 holes in the process, the atom itself is unstable as the bonds are incomplete on two sides(2 electrons from silicon form 2 covalent bonds with 2 electrons of zinc, while the remaining 2 electrons of silicon do not form any covalent bonds). Similarly, if we take a group 16 element, like sulphur, now although sulphur contributes 2 electrons in the doping process, what happens is that it makes the entire crystal lattice bulkier. Each atom instead of having one electron loosely bonded to the nucleus now has 2 electrons bonded, hence making the structure again unstable. That's the reason why we prefer to choose trivalent and pentavalent elements as sometimes it is essential that we settle for the less in order to achieve more.
2 or 6 elements have more ionic properties and hence introduce more defects. Making a free single-charge element is easier than a higher carrier dopant