R.K.
Unless you have this all in detail in a court order there is not much you can do. If you don't have one then don't let them go that day. If you don't have a court order you need to get that going to protect all of you.
Edited: if it's in the court order that he must bring them home by a certain time and he doesn't you pick up the phone and dial 911 because that is technically kidnapping.
I had to have our court changed to include that our child is to attend all events (parties, play dates, sports, cub scouts, etc) that fall on dads weekend. Something you might want to keep in mind for the future.