To be physically fit is just a small aspect. You can be a beautiful physical specimen, but you're empty as far as what it takes to be a person, and that shows up real fast.
A good example of how it must have been is today's world of conducting, which is still utterly dominated by men, and the prejudice the few female conductors have to battle even today is astounding.
When I talk to people outside the beltway, I don't think people are that divided.