Caring for children is a dance between setting appropriate limits as caretakers and avoiding unnecessary power struggles that result in unhappiness.
You have to listen to the people who have a negative opinion as well as those who have positive opinion. Just to make sure that you are blending all these opinions in your mind before a decision is made.