Olive Tree Leaf and Stem Spots

Hi All, I must confess I am not someone who knows much about gardening, but I am an enthusiastic idiot. A few months ago, we purchased a small olive tree from a good garden centre. We have it in a pot in a really sunny spot, albeit it was in its original plastic pot inside a ceramic one. A week ago I noticed rust coloured circles on some leaves, and suspected it may be due to giving it too much water and the plastic inner pot was causing poor drainage. I have since removed in from the inner pot and potted it up. However, the rust spots seem to be spreading and it looks to now be affecting a branch as well. I have done a few searches and it could be a fungus, but I am not seeing anything quite like that pictured. Help with diagnosing the issue and remedying it would be great please? 👍Many thanks