More than two school of thoughts. As a Physiologist I believe in a strong link (as u have mentioned). However it is not agreed upon by works of some famous psychiatrists who prove that ED can be linked to a particular female in humans (fear, anxiety, etc.) and it is cured 100% with another. As a Physiologist I can safely say that again it is pheremones and not 'fear or anxiety'. Anyway this is a wonderful topic that needs more work.
I am sure you have gone through these articles:
1: Motofei IG. A dual physiological character for sexual function: libido and
sexual pheromones. BJU Int. 2009 Dec;104(11):1702-8
2: Motofei IG, Rowland DL. The physiological basis of human sexual arousal:
neuroendocrine sexual asymmetry. Int J Androl. 2005 Apr;28(2):78-87
females themselves are quiet vulnerable physiologically and emotionally. They may be affecting emotions and behaviour of a man with who they have relations, but I see no such bodily factor that can create such impact on man.
You must undertake cohort studies of men, may be that's challenging but may be that can help identify some common patterns like erectile dysfunction in hostel boys where they stay together, Casanova type boys or men usually surrounded by female for long day hours, or gays, lesbians, etc. You will surely find psychological differences in them. Psychology also plays a vital role in erectile dysfunction.