This kind of charge reveals a good deal about the personality of the people who make it; to impute such motives to another man is to imply you're harboring them yourself.
Only two things are infinite, the universe and human stupidity, and I'm not sure about the former.