Fungi do grow on all organic materials, the speed of growth might differ between substrates. Moisture content (water activity) and temperature are very important factors in this respect.
Fungi are widely distributed in nature , there is some factors affect their growth on food like moisture, temperature, oxygen supply, type of substrate or food, pH...etc.......
I agree with all of the answers above. Because bread typically has low water activity due to the hygroscopic nature of starch and gluten so the environment of bread is typically has too little "free" (not chemically bound) water for bacteria and parasites to grow and thrive. This leaves fungi (archeaea do not cause human illness, at least as far as we know) as the primary cause of bread going bad. Fungi grows slowly compared to bacteria but can survive in environments with higher water activity and more extreme pH. Sourdough tends to last longer than other types of bread because of the low pH. Breads high in fat or sugar also tend to extend shelf life because the fat replaces some water as a "moisturizer" and sugar lowers the water activity of the bread. So a fatty bread like brioche or a sweet bread like banana bread will likely last longer than a simple baguette. And as a final note, the other issue with bread that often occurs prior to molding is going stale. Certain breads are more resistant to staling such as rye bread, which contains pentosans which do not retrograde and harden into crystals the way starch does when it goes stale.