Olive tree - black and white leaves problem

Hi folks. Can anyone tell me what's happening to me olive tree. It's been on my patio in a pot for several years and is now about 4 foot high. It's never done amazingly well and I've never had any fruit but this year it seems to have a real problem. 

All the leaves are blackened with white spots randomly scattered. I took a leaf off to examine and the blackness is a kind of smutty covering on the top surface only and can be rubbed off with a nail. I'm thinking it's a fungal problem but I'm no expert. Can anyone suggest anything?

Cheers

James