Olive trees

Hi please advise me regarding my two rescued olive bushes. I have grown them on from virtually twigs for the last 6years and  they are doing well at at least 4 feet high now and kept outdoors and in full sun in the summer. Last year however I noticed some curled leaves and found very small greyish green caterpillar type bugs which were making holes in the leaves. I cant find any reference to these bugs and don't know how else to rid the plants of these pests other than picking them off almost daily. Please help!