There's always going to be someone out there... who doesn't believe in you or who thinks your head is too big or you're not smart enough. But those are the people you need to ignore, and those are the times you need to just keep doing what you love doing.
Much of what is called investment is actually nothing more than mergers and acquisitions, and of course mergers and acquisitions are generally accompanied by downsizing.