If you have no visitation set up through the courts and noone has custody...you just broke it off and seperated, I believe technically you could drop your children off for the weekend and he could decide not to give them back and there would be nothing you could do but take him to court and fight. You wouldn't even beable to call the police. That's what I've been told.
With the drug use minor drug use? How do you know it's minor? It's one thing to dabble around and smoke pot in college and it's another thing to be "grown up" and do drugs~in my opinion. I know a awful lot of men who claim to only get high every once in awhile but it's really alot more and they are really good at hiding it. (my 25yr.old brother) his girlfriend has no idea how much he actually does.
With that if your ex could prove to a judge that your children are around someone on drugs and mom isn't doing anything to stop it you could lose them to child protection or he could get custody. Now he would have to prove it, as in someone with authority would have to give your boyfriend a drug test. It's like if your boyfriend is abusing your children and you cover for him or just watch it happen but don't remove your children from the situation(move somewhere new) your just as guilty in the eyes of child protection.
I'm glad the new boyfriend is great to you and your kids but if he loves you or them he wouldn't do anything to jepordize your kids well being.