Plum tree - curling and brown leaves

We have an established plum tree that was here when we bought the house, and which provided a great crop last year. I haven noticed quite a few curling leaves (which internet research tells me is aphids and there is nothing I can do about it) and a few clusters of brown leaves I can't explain. 
Any ideas?