I've never found it made the slightest difference being a woman - though there is a sort of feeling that as you get older you're not so interesting.
People who say they don't care what people think are usually desperate to have people think they don't care what people think.