Improving drainage in lined raised bed without digging it all out?

The bed is in full sun, 2m long by 0.7m wide, timber and lined with plastic with holes cut in the bottom. It sits on builders scree leftover (new build property). I think there is a drainage problem as the soil is consistently damp a couple of cm down and I've lost a few plants already which seem to turn brown at the ends (like a houseplant which has been over-watered).  For example, Jasmine has flowered but the flowers are turning brown and curling up. 
I've added more perlite and compost mix to the soil and taken out plants which seem to do better in a smaller pot (better drainage?). Would adding grit or even worms help? Or should i simply dig out the whole lot, cut out the bottom of the lining and add stones, then refill? I don't want to lose any more plants but I'm worried about hitting autumn and winter. Thanks for any advice!