E.A.
You need to use some behavioral "therapy" to get him to break this habit, or it will become a serious medical condition if he starts to hold it to avoid the toilet. Try sitting him on the toilet after every meal, about 10-15 minutes after he finishes eating, this "trains" the bowels to evacuate after consuming food. Make it a normal routine and I like the idea of the reward for going. You should never punish for potty-related behavior, only use incentives and positive reinforcement and a regular routine.
He's already showing signs of withholding, so it's really important you nip this in the bud now before it becomes more serious and he truly can't control it. That's a whole other bag of worms...