Luckily, the kids are young and so 5 years from now, will not remember your denighing him visits with his friend at age 2.
In my eyes, you could go 2 routes. 1, let your son and he be friends still on a limited basis, but slowly close that connection with them and do not be receptive to outings and get togethers. OR
You could cut everyone off and slowly start to reject phonecalls and invites for gatherings, etc. But, if you do this, have an excuse because they think you are friends and will demand and expect that you will have a good reason for blowing them off more and more lately.
The important thing is to do it slowly. I'd start by telling them a lie about an older family member or close friend, neighbor, etc. of yours or hubbys needing help with their life due to their health and requiring much of your time. That way, when the visits and calls start slowing down on your part and they ask why you didn't call back, or why you can't get together, it was because of your sick/needy person. If you set it up ahead of time (it could even be a fake person), it'll be easier for them to accept your blatant attempts to distance yourselves. You just won't have time for it all. Then,...pretty soon, you'll be away from them and you will all have the understanding that you all just "grew apart". No harm done and no hard feelings. I'm all about not hurting feelings.
I did this with a gal whom I met at ECFE too and her frienship just came on too strong for me. She thought we were best friends or something even after only knowing her for a week or 2 and I just didn't feel the same. (I think she didn't have friends and was a bit desperate so she clung to me calling me 3-4 times a day). I started slowly and I even admitted to being a schmuck and apologized for not calling her back right away, but I was sooo busy taking care of my godmother (whom I haven't seen since I was 5 but didn't tell her that) that I've just been so busy and haven't had time to call her back. Eventually, she got it and stopped calling all together. (I think it took about 6 months.