What to plant on sloping edge near steps

Advice be appreciated please. What should I do about the edge of the steps we put in last year and the sloping clay bank. 
My garden is at the back of a disused quarry, on a hill, clay soil was over grown with bracken but I have had it covered or I've cut back for over 10 years. The ground has an unbelievable amount of rocks whenever digging anywhere.  The garden is to the elements too.
Any idea of what I can do to stop the sides eroding and what would make a impact quickly. I don't want to plant ivy as it's too difficult to control.