D.P.
You've gotten a lot of great advice so far.
Anyone who as a third party to "sits you down" and tell you about things that are negative (you need to "protect my kids and the person I'm with is not a good person") cares enough to do so. This is more than this person doesn't care for him, etc.
PLEASE move out, keep your kids safe, keep them the priority in your life. You can date people, but please don't drag the kids along for the ride, they've already had to adjust to leaving dad, not this guy...will there be no end?
Do what YOU need to do to suppoet yourself and your kids, and make a life for you and your children. Please keep the kids out of your dating life (and not in a live-in situations agian) until you have a date set and a ring on your finger. After all, first things first! Best of luck.