I'm with you emotionally, however, legally -unless you have right of first refusal- your ex can leave them with anyone, for any period of time, as long as its on his time.
If it happens a LOT it's grounds for changing the custody agreement, to an extent.
Just as an example, my ex has my son in daycare from 5am-8pm, or from 6am-6pm. Ive been informed no court (here) will regard that as grounds to change custody, because every parent has the right to work. If he were military and deployed, or travelled for business, then it would be grounds, but NOT since he's local.
These things are 2 way streets. While I HATE missing out on my son, I can also choose to have him spend the night at a friends or his grandparents, or go to camp, because I ALSO have the right to say where and when my son is at during 'my' time, as long as I'm actually around.
So while SM has no rights, dad does. Living in divorce and custody land, the fact that yours has a blood sibling via SM will make it even less likely that the courts will intervene on an infrequent trip.
But if these trips are the norm? Then you have better legal legs to adjust your parenting plan. Doesn't mean an awesome lawyer or a sympathetic judge couldn't change things. Just means you gave a better shot, legally.
It's a LOT easier to change anything by agreement, than in court, though. So if you can work it out (in writing) ahead of time.