Never continue in a job you don't enjoy. If you're happy in what you're doing, you'll like yourself, you'll have inner peace. And if you have that, along with physical health, you will have had more success than you could possibly have imagined.
No one should negotiate their dreams. Dreams must be free to fly high. No government, no legislature, has a right to limit your dreams. You should never agree to surrender your dreams.
As an actress, you're already disregarded for a lot of the parts by the people who are setting up those shows. You don't need your agent to be doing the same.