Well, it's our own children that drive us the craziest, right?
Sounds as if your brain is going in several directions at once.
Only you can answer this: is your frustration at a level where you need to seek some counseling to get your feelings sorted out?
Let's say it isn't. OK, try this:
Stop what you're doing, and look at your son. Just look at him. Pretend he's somebody else's kid. What do you notice about him? What do you like? What would you miss if he weren't ever around any more?
Do the same thing once an hour. (Sounds like a doctor's prescription.)
Do you love this boy? (Of course you do.) Then tell yourself so. "I really love this boy." Then tell him so! None of this "I love you, but..." - just "I love you," with no strings attached. Smile at him every time you see him. Every single time!
Why am I telling you all this silly stuff? Because it sounds to me as if your brain is racing with all sorts of things that need to be done and so little time to do it in and so little cooperation from the family and... you've stopped focusing on the fact that... you're talking about a real live, very special person! Put a big red sign in your brain that says STOP! There will always be lots of things to do and ducks to get in a row, but there will only be ONE of this boy. Wow!
Next time you're with him, really listen to him. Talk TO him, instead of talking AT him. This will bring your focus where it belongs and the patience may build.
It's a good thing that your son is growing older. There would be something dreadfully wrong if he didn't grow up. Time goes faster and faster. Slow it down by focusing on the people in your family as much as you can. The other stuff will get done, but it doesn't have to take you away from what's MOST important.
Hope this helps a bit. Hope it even makes a little sense. I struggled with this same situation.