I suppose if I had my time again I would refuse it and stay at Fulham because I thoroughly enjoyed my time there, and secondly I would have taken it on my own terms.
Instead of worrying about what people say of you, why not spend time trying to accomplish something they will admire.