Lomo chilli

Ive had a chilli plant for a couple of months and it had small chillies on when purchased and they are still the same size but today a couple of buds have flowered so my question is should I have plucked off the chillies that are obviously not going to do anything except shrivel or leave them on the plant