Planting wildflower plugs in wormcast smothered soil

I have a perennial wildflower meadow covering half of what used to be the lawn. The soil is incredibly poor, sandy soil, crammed with stones. In places the soil seems to be barely an inch or two thick and establishing a lawn proved futile, no matter the effort I put into it,  hence the initial reason for trying to establish a meadow in Autumn 2019. I sowed a sandy soil/drought tolerant seed mix into the heavily scarified ground and the results this year have been good. So far this year I've had lots of Bird's Foot Trefoil, Ox Eye Daisy, Ribwort Plantain (which actually looks good in this setting, despite being a pernicious weed for most gardens!), Wild Carrot and Yellow Rattle. Two or three Vipers Bugloss have grown and about four Common Knapweed have emerged. The picture shows it at its best a few weeks ago and doubtless it was helped by May's rain. 

I want to improve the meadow by planting wildflower perennial plugs. Hopefully more Vipers Bugloss and Knapweed etc will grow but I'd like to give them a helping hand. However, during autumn and winter the ground is alive with earthworms and literally smothered in their casts. These can be quite large - certainly large enough to smother a wildflower plug. Given how many casts there are, would it be better to plant the plugs in the spring, once the casts die down, or are the plugs tougher than they look and able to survive such an onslaught? 

As I understand it, autumn is the best time to plant plugs, so their roots can establish before dry weather sets in. If the dry doesn't get them, maybe the wormcasts will. Does anyone have any experience of this and whether I should plant plugs in autumn or spring?