There are only a few places that I go where people recognize me anyway. I have to be at the right place, in the right city. It's not really that much of an issue.
A doctor must work eighteen hours a day and seven days a week. If you cannot console yourself to this, get out of the profession.