Okay, I have 6 kids and a LOT of other kids in my neighborhood some ok some really rough/rude with parents who don't supervise them enough and are always just kicking them outside to "play" until they have to check in (usually a LONG time later so the parents can have time to themselves). I have been on both sides of this issue. (And my oldest is only 9 - so this is still a "current" issue for me).
First of all, I *never* let my 3/4 year olds out of the yard. Unless, they are invited over specifically by a parent whom I trust and who I know will be watching them the whole time. And whom I know will bring them back to me or hang on to them until I come to get them. No one is allowed to even cross the street until they are 5 (barring a walk with parents, etc)- even directly across to play with friends.
Second, I realize this is tough behavior for your son to take in stride, especially since he is an only child and only used to being around people who all like and value him and have their "grown up" problem solving skills already. However your son is a toddler, and he will meet all types. You can't change people, and at this age - kids are super nice and super mean with barely a hair's breadth inbetween. It's really pretty normal. I think it's unfair to label this kid a "bad" kid or a "bully" when he's obviously got a lot on his plate to manage as far as personal relationships with people go. Overstimulation aside, this poor kid has to watch others playing with his stuff all day. Imagine if you had to share literally everything you owned with everyone all the time. (I realize that as a parent you share most of it, but I'm talking about your personal, private things being shared with anyone and everyone who showed up uninvited and unsupervised) He's just a kid like any other kid. At first, I reacted to some of the kids in our neighborhood like they were bullies because they were rougher than a lot of the other kids we had run into. BUT, at some point I realized that wasn't fair to them - they are just kids. When they are in my yard, I try to remember that no matter how bad they are acting they are just kids - and I tell them when they are breaking the rules of my yard nicely. If they don't stop - I send them home for awhile. I also make an extra effort to say something nice to them the next time I see them after that. Just as a reminder that it's their behavior alone I have a problem with. This has seemed to earn myself, my kids, and their things some respect.
However, occasionally my kids will go play with the bullies and then come home whining about the rough play. My feeling is that if you know the kids are playing a rough game, and you don't want to play rough, be smart enough not to play. That's just good decision making. They are either willing to put up with the rough play and try to use their people skills to make the situation better, or they're not. Of course the kids in question here are older than 4, but the same principle.
All you can really do is remind you kid how to act, maybe help him have some compassion for the bully kid (you'd be surprised what being compassionate - while not allowing yourself to be a victim - to a bully can do for a situation), give him the tools to stand up for himself, and call/talk to his mom. See how she feels about having all those kids in her yard. Ask if there are times that are better for your son to play, or if she minds the little ones being there at all... OR, see if her child wants to come to your house to play one on one (sounds wierd I know, but some kids this age are TOTALLY different and MUCH better when they are playing one on one) to help him and your son kindly resolve thier differences and build a friendship where you can supervise their sharing and interaction. Ask the cousin's parents directly if he can come over and play some time so the boys can hang out. You have options, and I don't think teaching your son this kid is "bad" is necessarily the best one. Of course, teaching him that he may want to tell this kid that he doesn't want to play with him if he's not using his manners is totally acceptable. Just explain to your son that it takes some children longer than others to learn good people skills, and only children typically learn them fastest because they have exclusive contact with the best role-models - their parents. I'm sure this kid will turn around eventually and you may find that after you explore the situation with his mom, that a balance is struck all on it's own.