I'm guessing it's because the ligand experiences too much electron repulsion or proton repulsion from the chromium to insert them close to the 3d-orbitals which are close to the metal nucleus. Is that correct?
Hi, For octahedral bonding, ML6, the sigma interactions with the L ligands do involve mixing with the d-orbitals (just the Eg symmetry though). See for example Ch 10, page 367 in "Inorganic Chemistry" by Miessler, Fischer, and Tarr. Symmetry and orbital energies define what can mix and form bonding and anti-bonding orbitals and what orbitals remain atom-centered (non-bonding).