Citrus plant advice

 Hi Guys

I have a Kumquat plant that is about 2 years old.

When I purchased it it had fruit on but has never produced fruit since the first year.

I read up about them and took the advice to bring it indoors for the winter. It was in a bay window that gets plenty of light but as UK winters produce little sun it was no doubt lacking this element!

Throughout the winter it lost all its leaves and I thought that it was probably dead or dying! Not wanting to give up on it, I managed to get use of a greenhouse during most of this year and it made a bit of a comeback to the level it is at now. 

Unfortunately, I had to move it out of greenhouse 3 weeks back. Not wanting to shock it by now putting it outside, I have had it residing indoors again, however, a couple of leaves have already fallen off! There is a new shoot that has come through over the last month or so.

I am fearful that I am going to be back to square one again with a plant with no leaves. I am in a dilemma of keeping it indoors or putting it outside. I live in Jersey in the Channel Islands.

It has a number of fungus gnats residing in it which I am trying to clear using biological soap, vinegar and water in little bowls. I have read that the gnats will not do the plant any harm, however, they are annoying!!

I have attached some photos of the kumquat in its current state.

Any advice would be greatly appreciated!

Many thanks

Chris