you can't set those limits, but the courts can... hopefully you have a good custody agreement, but even then you usually have no control what your daughter does while at her father's house... just like he has no say about how your daughter spends her time at your home. Unfortunately it's part of living with a broken family, you don't have complete control anymore.
But if you feel she's in danger then you can document & get all the proof you're able to and file to ammend the custody/visitation - maybe get a right of first refusal included for when he's not available during his scheduled time.
I saw this from the other side - even though they have joint custody, my husband's ex has residential custody. When i met my husband, his ex hated my guts & didn't want me in the picture (I don't blame her, she didn't know me and felt threatened, that's only natural)... she played a lot of games with his custody, even tried to keep their daughter from going on vacation with his family when she found i was going. In the end my husband was able to show her how rediculous it was - that she couldn't control what he did during his time, just as much as he couldn't control what she did during hers. That she wasn't in danger & that they are BOTH her parents. And that they could discuss it respectfully, and its always good to have open communication, but the parent who's time it was had the ultimate say on how they spend their time with the child.
If you feel you should have say in what happens during his time, then he should also have say what happens during your time - not saying that to be disrespectful at all, just to give you some perspective, it does go both ways. I'm assuming you don't tell him every time you have someone watch your daughter, and he may not like everyone that you bring into your daughter's life either. (My husband's ex does some things my husband doesn't agree with at all & has some crazy discipline methods, but there isn't much he can do... its nothing he could take to the courts... & at ex's home its her rules, just like at our home we have our own rules).
Doesn't make it right, what your ex is doing - a bunch of women in & out isn't a good choice on your ex's part - my husband waited 3 months before introducing me to his daughter for that exact reason. And my husband tried his hardest to never need a babysitter. But life happens... and people don't always make good choices. That's where you come in - to give her an example of how a "normal" adult acts and what good relationships look like.
If he's leaving her with a person who is drunk or drinking, that's something to make a bigger deal about, but it's a hard thing to prove... some people act foolish in public, but have a bit more common sense when it comes to watching children. I actually know someone just like that. She can be really rowdy & dress really inappropriately, but she's a nanny by day and is wonderful with children! Hopefully that's the case.
Not saying you shouldn't look out for your child's safety, but no parent is perfect - not him, not you. Unless you really feel your daughter is in danger, you can only try to talk to him and express your concerns and hope he understands. Kids are smart and they're tougher than we give them credit for. We can't protect them from everything, but its our job to teach and prepare them for everything they'll face in their life, with good morals & good coping skills.
Sry for the long response, but hope it helps even a bit. I know how difficult it can be.