My brother was an improviser. He's now a lobbyist, but he used to perform improv in the city when he was in high school, and one of the funniest guys I know to this day.
Instead of worrying about what people say of you, why not spend time trying to accomplish something they will admire.
If people are going to complain about stereotyping, it's as likely to be Italian-Americans as gay people.