Help needed to save orange tree!

can anyone give me any ideas about what could be wrong with my orange tree. I was given the tree last summer and it lived quite happily in my orangery until November when I brought it indoors as there is no heating out there. It had already started to drop leaves before I brought it in, and it continued to do so for a few weeks. The leaves were going crispy and brown before dropping off. Since then I haven't had a problem with dropping, but it doesn't look too happy and I'm not sure what's wrong. I only water when the soil dries out, and have just this week put it back out into the orangery in the hope some sunshine will perk it up. It's not growing any new leaves, and the ones that are on there have gone a bit limp and I think may start dropping again soon. (The fruit has been on there months, should I remove it?) Any suggestions would be most welcome!