D.N.
I would get an exterminator unless you are willing to take the risk of finding their nest. We had a paper wasp nest in a tree a few years ago. My brother was over and being the risk taker he is, climbed the tree and took it down. It was empty but they are not as bad as regular yellow jackets.
The one thing I hate about bbq in August.