Instead of worrying about what people say of you, why not spend time trying to accomplish something they will admire.
“Whoever I am, or whatever I am doing, some kind of excellence is within my reach.”
“Dullness in matters of government is a good sign, and not a bad one - in particular, dullness in parliamentary government is a test of its excellence, an indication of its success.”