The two things that can hurt you are if you need money or if you need fame. Those are the things that can be your Achilles heel. But if you don't need money and you don't need fame, then you're free.
“Our constitutional liberties shall not be sacrificed in our search for greater security, for that is what our enemies and all enemies of freedom and democracy hope to achieve.”