In men the incidence of breast cancer is lesser than women because although the breast tissue in both, male and female is similar, men have mainly fat and fibrous breast tissue called stroma, and they have fewer ducts and lobules. No doubt women have many more breast cells than men, but the main reason is that their breast cells are constantly exposed to the female hormones estrogen and progesterone made by the ovaries (known as endogenous estrogen and progesterone), which promote cell growth. Being exposed for a long time and/or to high levels of these hormones has been linked to an increased risk of breast cancer in women.
Men normally produce small amounts of estrogen, but in certain conditions like gynecomastia which means a condition where the male breasts are abnormally enlarged in response to elevated levels of estrogen, there is a risk of developing male breast cancer due to high levels of estrogens. Most breast cancers in men are estrogen receptor-positive (i.e., they have proteins on the surface of the cells that can receive and transport estrogen to the interior of the cell).
Estrogen is one as mentioned by Malcolm, also in modern times as compared to earlier times, women are using contraceptives and hormonal analogs to mess with reproductive cycles and for preventing pregnancy. Every company wants to sell their such products but they will always hide the harmful effects of these agents which prevents pregnancy. Our mother generation, women never used these. All these agents affects mamary gland, uterus, ovary. Companies will never do research towards the harmful effects of these agents.
Breast cancer is a hormonal dependent cancer. Female hormones as estrogen and progesterone play a crucial role of its development, except for triple negative ones (basal like), where genetic predisposition (BRCA 1 and 2 genes) are more encountered.
In males, this female hormones are very low, it can explicit its rarity in male persons.