If the work is pure then you have to think it could be understood. If it is not understood it doesn't mean that your work is not accessible. It doesn't worry me, but, of course, I would be pleased if people liked my work.
Writing can be a very solitary business. It's you sat at a desk typing words into a computer. It can get lonely sometimes and lots of writers live quite isolated lives.
If somebody pisses me off and I understand it ain't personal then I'll go to another place and I'll meditate.