I would leave it there, I agree with one of the posts above that it can be removed anytime if it gets too big/if needed.
But this is a cultural difference, I am originally from a country where people are used to many tall trees much closer to houses.
The point about selling and surveyors is valid but if you always consider this with everything you do to the house, it will never be a true home.
It's "don't plant this plant, some buyer could think it is a japanese knotweed", "don't plant this tree, someone could think it is too close to the house (no matter what tree it is and where)" or "don't make the garden too nice, someone could think it is too high maintenance/too difficult to get back to lawn/gravel", this thinking sucks out all joy out of owning a place... just my opinion.
Posts