Ask yourself: If there were no kids involved -- if they had none, if you and your husband had none -- would you want to spend time with your brother and his wife, as adults with adults?
Do you like them personally, if you take kids and that feeling of "cousins should know each other, aunts should know nieces and nephews" out of the picture?
Do you have any interests or activities in common with these two adults? Would you choose to go to the movies with them, have them over for dinner, wonder how they're doing, want to check in with them -- IF they were not relatives whom you already know?
If you wouldn't hang with them as unrelated, childless friends, then the only link you're feeling is a sense of some family obligation that clearly isn't reciprocated from their side. Sure, it would be nice to get to know their kids, and as your kids and theirs get older, and SIL and brother perhaps get some reality checks about the fact that all kids will get sick, etc., you can try again with them in the future for more play dates for the kids. But "baby love" only goes so far in binding adult family members together.
Find friends who aren't relatives but whom you like, have things in common with, want to DO things with that aren't entirely kid-centered. Sad to miss out on nieces and nephews but you do see them, if rarely. Meanwhile, it's even sadder to miss out on real, close friendships for your kids and yourselves while you're pining for what sounds like a closeness that is never going to happen.