Fungal infestation?

I have made a flower bed over the site of an old Lime tree which we removed three years ago because it was rotten. Since then I have added homemade compost and planted a variety of plants. The soil has slowly become hard and formed a crust about 5cm thick, it is almost wood like in its density and very difficult to get through with anything but a spade. Tiny mushrooms have appeared all over it and it is slowly spreading. It is now approximately 1 meter square. Last year I removed as much of the contaminated soil and replaced it with compost. The older soil around the new compost is now hardening and has mushrooms and I’ve noticed the newer replaced soil is also starting to harden again. Should I treat the soil somehow (I only use organic products) or should I not worry about it and leave it as I’m wondering if it will just naturally sort itself out? Any ideas or suggestions would be most appreciated.