I always try to go with my Mommy gut on this. Even if he is feeling better I know that had I sent him in it could have been just the opposite if I didn't keep him home.
My son is a tough read. He has had a fever of 104 and still playing and acting normal. It happens and I wouldn't feel guilty. Maybe you both just needed a day to rest and recupe.
As far as the hubby, mine is the same way. Just about every time he acts nasty about my decision to keep him home. I told him the last time I had him stay home, "Look, I take care of drop off, pick up, sick calls, doc visits...he doesn't miss school unless it's necessary! Back Off". He actually told me later, "I'm sorry, your right".
Take care, hope everyone is on the mend.