cayenne pepper pest problem

HI everyone, I have a few cayenne pepper plants in the greenhouse but the leaves are being decimated by some sort of pest. loads of holes eaten in them. I a hoping the peppers would ripen and i could get them off but the plants are really struggling with the pests now. I pulled off a few caterpillars earlier and think its them that have been doing the damage.

Question is, can i spray the plants with anything organic ? I have seen stuff online about spraying with a garlic/online/chilli spray or even adding a small amount of liquid soak, but will this not affect the chillis on the plant ? Not sure soapy chillis sound appetising haha! 

I would appreciate any help on what could be done. 

thank you,

Jamie