It's none of your business what he does in his own home. That's absurd to have had anyone tell you that you could control him like that.
If you and he don't get along then the further out of his life you stay the better off you will be. If the kids don't like what's going on over then there THEY need to talk to dad themselves. Depending on their ages they could even go to court and have their visitations revised, reduced, or removed. If they don't have a problem then they don't need to talk about what's going on over there with you, at all, okay?
How would you feel if they were going to dad's house telling everything you're doing at your own home? If you were doing this or that and it wasn't really any of his business.
It goes both ways.
I had a co-worker that went through a nasty divorce. He divorced her because she got "saved" and went off the deep end. Refused to allow him to even take his own kids to his parents house because they smoked. So when they went to court she put in the divorce decree stuff from her attorney, that the judge had to rule on, that smoking around her kids caused them to get sick and she requested that the judge not allow her kids to be around smoking.
The judge said okay so the dad requested that any place where smoking was allowed would be off limits. So guess what? The kids don't get to go to the county fair, they can't go to flea markets, they can't go to public rodeos or anything because mom didn't want her kids to be around her ex's family and came up with the smoking idea and thought she'd make sure "her" kids wouldn't ever be around "his" family. And that would hurt him.
In the end it was her and her kids that paid the price more than anything because one of their favorite family activities was going to like county fairs. The girls also couldn't do any performances in dance that weren't in auditoriums. If they danced at an Oktoberfest event they could go, smoking would be allowed on the grounds, if the girls were going to dance at an outdoor event like a Renaissance festival they could go, smoking was allowed on the grounds, by trying to be too involved in the dad's life with his children this mother lost out on time with her own kids.
That's my point. Leave him and his household alone and just focus on your own. If the kids come home and tell your that she's abusing them or neglecting them and they aren't old enough to tell dad about it then you might want to figure out what to do at that point.