Fig tree

Our fig is on a south facing wall in a brick trough, so has restricted roots.  We water it in dry weather and good crops of figs start forming every year - but when they get to half the size of my thumb they fall off.  The leaves look well and there are plenty of new shoots so I don't know what's wrong.  Any ideas?