wasps-on-my-plum-tree

My Victoria Plum tree's leaves are covered in tiny holes but I have a bumper crop of plums compared to last year, however, the tree is swarming with wasps.  Are the wasps eating the fruit or aphids on the tree and is there anything I can do to get rid of the wasps as I can't even attempt to pick the fruit at the moment - luckily I do not think the plums are not quite ready for picking.  Thanks