The reason is more thermodynamic than structural. The additions alter the free energies of the different phases, which in turn affects the transition temperatures between the phases.

I found this link as well:

https://www.quora.com/What-is-difference-between-austenite-stabilizers-and-ferrite-stabilizers

Why certain alloy elements cause certain effects is trickier to answer as it could be structural, e.g. impurity size and location, and/or chemical, e.g. changes to bonding type and magnetic nature. Even if you use atomistic calculations, like DFT, the end goal is to compare the relative energies of different structures.