Does your husband ask? Like, "Hey, tomorrow do you mind if I go to the shooting range?" "can I leave the baby with you while I go to the gym"
Or does he just make his plans and assumes it is okay, then leaves?
Basically, you need to say "No", set limits and boundaries like he gets his solo things to do, then you switch off and you get your EQUAL amount of solo time. If you don't say anything, how is he to know.
OR, don't wonder about it, just do like he does and once he's awake and up and around (like on a Saturday), grab your keys and announce YOUR plans, and then leave. (if that's how he does it)
Or just sit down and make the ground rules and agree to amount of times of "me" times you each get.??? Hope that helps!