I'd recommend keeping the diaper bag always stocked with the exception of bottles/sippy cup and snacks. Otherwise, ours is always full of the things we think we may need.
We're both working parents, too, and cherish the time we get with them. He's getting to an age where they're really curious and SO much fun.
We keep a bag in the car with toys that we can take into restaurants, to the park, etc. to keep both of our kids entertained - age appropriate, of course, and a change of clothes in the trunk just in case.
Even though we're tired, too, and spend our weekend running around doing errands, we will stop at a park, the library, etc. and just let them run. At that age, you really need to supervise.
We went to DC over Christmas to see my family. We thought our 3.5 year old would love the Smithsonians as he's been watching the Night at the Museum movies. He could have cared less.
So, as long as you're interacting with him at this point in time, it really doesn't matter where you take him. He'll probably be most interested in the least important things.
Good luck!