A.L.
I think you should go out, definitely. Every mother needs a break and every child needs to know that parents need adult time. Especially if you are a single parent -- you really deserve a break!
My daughter will try guilting us once in a while (we only go out like 4 times a year together and I go out alone or with a friend about once a month). However, I try not to let the guilt bother me b/c I'm home with my daughter all day every day and I need a break and some adult time. I know she is perfectly fine (either my husband or my parents watch her), and it is good for her to be taken care of by others.
I know it is hard, but try not to feel guilty. Having some adult time is good for you and will make you a better mother.