I doubt that he would have left his "affair phone" with you if he was having one (what if she called it?) I would ask him, lovingly and NOT confrontationally, what has made him so distant. Perhaps when you're both in bed, before going to sleep. Tell him that you're worried about him, and that you love him and miss him, and ask him about that number in his phone... (again, make it a safe environment to talk to you).
There are a lot of bitter responses here from people who have been cheated on, so they assume the worst. He may or may not be having an affair. It can be awfully insulting and hurtful to be wrongfully accused, but you should be able to talk to him and tell him your concerns and feelings. Men have feelings too; when my husband doesn't want to talk, I snuggle up to him and touch him and won't go away until he does open up.
Dr. Laura has a wonderful book called The Proper Care and Feeding of Husbands. It literally turned my marriage around from being one of much anger and hostility, to being one of much joy and bliss. It details many cases of men who cheated, and WHY. GOOD men don't cheat unless they aren't getting their needs met at home. So often, husbands take a backseat in the family once the children come along; they miss their wives and need attention and affection just as much as women do.
Anyway, don't assume the worst until you have much better information. And if there is an affair, is it because of low character (and he'll be a repeat offender), or is it because he's being neglected at home? Don't give up yet.