M.P.
Ugh, sounds like my ex. Do NOT let him get to you! he's being a complete, jackass, jerk, prick (scuse my language) about it & maybe even starting to feel bad that he SUCKS as a father/dad and he's putting that on you. don't succumb to that crappy feeling, as most of us would because we're human & nice people! :)
continue forth w/what you know is right which is mtg w/him first & YOU deciding if it's a good thing for him to be involved. my ex has supervised visits now and the ball IS in my court - and i love it. i'm there at every visit (due to other circumstances), but still, maybe try it like that at first. (i'm pretty sure you wld since your baby's 3 & NEVER really known that guy - ugh). anyway, my only point was, he's trying to make you feel bad for HIS shortfalls. YOU'RE the one who's been raising this sweet boy from day one, and it IS up to you who you involve. you have to make sure it's a healthy relationship, and will benefit your son. if he's gonna flake or keep hurting your son (by not showing up)...that's something to consider when thinking about future visits. just my opinion. i'm right in the thick of something very similar & speak from experience, i promise. i wish you strength & patience w/yourself lots of love! good luck mama! :)
(btw, my boy's 2.5 yrs)