Large but not very successful post build lawn

The builder roughly spread a mound of what they said was pure topsoil from their excavations after they finished the house. When we came to rake it out it was full of stones and even builders' rubble but we cleared this as much as we could, and rolled, raked, and etc, in late March. In early April we sowed plenty of good quality strong lawn mix from Emorsgate (where we got wildflower meadow seeds from for another part of the garden). The current lawn mix has finer bent and fescue grasses plus some dwarf perennial ryegrass and smooth-stalked meadow-grass. It took a long time to get going in April/May and as it did so a huge lot of weeds came up too and a lot of completely empty patches showed up despite our very careful sowing and watering. We have taken out the biggest weeds by hand and have just mowed for the first time after rolling gently last week. The lawn does not look good, very patchy and full of small weeds we don't want to go for by cutting too low. There are so many stony earth patches we feel we shouldn't resow now, but wait till autumn and live with the ugliness. What should we do? And should we use weedkiller in the autumn? I'd rather not but there are so many at the moment