citrus plants

Have a citrus plant for about 5 years potted.  Each year I stand it outside where it always does really well as soon as I bring it into the porch or cold greenhouse at this time of year the leaves and fruits start dropping.  I brought it in a couple of weeks ago and the leaves started yellowing so I put it outside again.  Has anybody had same problems.  I am very tempted to leave it outside and cover with fleece when the temperature really drops. Being in Corn wall we don't usually get much frost. Have about 12 really good fruits on it at present.