Wasp Nest

I have just see what looks like a wasp fly underneath a small pot (about 6inches in height) that i have a tomato plant in carrying a leaf.  Does this mean i have a wasps nest being build in the small pot?  if so what should i do.  I'm only in a small yard and don't want lots of wasps flying about but am a bit reticent to harm them.  My other half has suggested taking the plant out and seeing what is there to get rid but I'm scared if i move it there will scent left behind and the wasps will get angry.  I don't know what to do, Help!  Any advice would be great.