L.R.
I did not go through this myself but a close friend did, though not to the degree you are -- her ex soon realized it was best for their son to maintain his activities while on "dad's time."
A lot may depend on the ages of your children. Your ex needs to realize: By the time kids are in grade school, they are often involved in activities in which children who do not participate fully -- the ones who miss the weekend sports practice, or the dad's-evening dance rehearsal, etc. -- are eventually thrown out of the activity. Activities increasingly become commitments once kids are old enough and that happens pretty fast especially in sports, dance or music (group) performance. If for instance you have a child who is in a sport, eventually a coach is going to say, "If Kid can't be at the weekend practices he needs to find a team that doesn't do competitions. It's not fair to other players if one player is consistently not at Saturday practices and games--they won't know how to play as a team." I know that if my child were not at all her Saturday rehearsals for ballet performances, she would quickly be removed from the studio's performing group, and membership in that group is hugely important to her.
So dad needs to be told that the "not on my time" attitude will affect the kids and stifle their ability to progress if that's what they want to do. It's a bigger deal once they are about third to fourth grade when commitments are expected from groups and teams. Even if it's an individual thing like tennis or a piano lesson or some other solo activity, if they are in lessons, a teacher will soon drop them as a student if they consistently are missing classes.
He needs to find out that the KIDS, not you, will end up resenting this deeply. Are your kids old enough to tell dad for themselves how the feel about their activites? Are the old enough to stand up to him and tell him that they know they will be asked to leave something they enjoy if he refuses to take them? If so, it may be up to them. You could always find a way to have a coach or teacher let him know (but without letting on that you intervened -- sounds like he would just get angry if he knew that). Maybe if he takes a kid to an event just once and the coach or teacher says, "Sorry to hear about your family situation, I wanted you to know that we expect to see Kid on weekends and weeknights and hope you will volunteer with us...." or whatever...maybe that would influence him?
Sadly, he is just using the kids' activities as a way to control and upset YOU. He is not thinking of the kids' needs at all. They are individuals who have their own interests and if he is going to be their dad, he needs to support those interests. But his focus is on himself. He is claiming that he wants every second of "his" time with them so he can be a dad, but being a dad means letting THEM be themselves -- even on his time. His time is their time too.
My friend went through a lot of that sort of thing -- her ex being very possessive of "his" time with their son. Once son was a bit older, the son was the one who told his dad, I intend to do this thing and if you will not take me I will call my uncle or someone who will drive me there, because my team is depending on me. It never came to that but it worked on the dad.