Grey spots on my Olive tree

Hi

I might have an issue with my Olive tree: small grayish circles and some early leaf drop. I've got a feeling it may have something to do with it needing a prune and increasing air circulation, but wld appreciate some good advice please!

NB, I don't think the container it's in is large enough, so could this be affecting the tree too?

Many thanks!