One reason could be oogenesis sensitivity. As you know, oocyte should accumulate a lot of proteins and RNAs types during oogenesis in its cytoplasm to be used after fertilization. So, oocyte has different steps such as previtellogenesis & vitellogenesis.
Egg has to support early development and often also to supply maternal determinants, usually first embryonic divisions rely on maternal molecules accumulated in the egg during lengthy process of oogenesis. Egg needs a long time to synthesize all needed materials. Sperm is small- delivers just DNA (sometimes also centrioles and mitochondria) so it can be produced very fast without the prolonged growth period