I haven't read through all the responses, but I can tell you my husband was exactly the same with my kids. Turns out he was in the end stages of renal failure, and we didn't realize it. We knew he had kidney disease, just didn't know it had progressed so much, and it really effected his ability to see rationally.
I would agree about having him get checked (if he is willing). Until things change, I would simply not leave him alone with the kids. You need to talk to him when you are alone, and tell him that he cannot watch the kids anymore. I know this is a huge inconvenience to you, but better than an accident happening.
I did this with my husband. He was not happy about it, but he agreed because I became so anxious, and he didn't want me to be super anxious. Either sitters watched the kids, or I did. He was never alone with him. Once he had a transplant and improved, then he began to change back to being rational. Now he is aware of what is and is not safe, so slowly I've allowed him to watch them on his own. So far so good.
I hope this helps,
B.