Wow!! Crazy!
My sister in law gave a baby up for adoption, and she still has A LOT of contact with the baby (it was an open adoption) and the adoptive family. I have met the family and the baby as has everybody in my sil's family/extended family.
I would recommend talking to a lawyer (which you probably already knew), and then if he has any rights, he should be a part of the baby's life, but I wouldn't recommend taking the baby away from the only mom and dad he has ever known.
One thing I would like to add. I am only saying this because the adoptive mother (of my SIL baby) had talked about this before. She insisted that both the natural mother/father and natural grandparents get counseling if they want to be a part of the baby's life. This is the only way she would let the open adoption happen. My SIL got counseling as did my mother/father in law. The natural father opted not to be a part of the baby's life at first. But, a few years ago (baby is now 5) he wanted to be a part of her life. The adoptive mother let him see the baby one time, but then insisted that he get counseling before he sees the baby again and before they explain to the baby that he is her natural father.
Hope this helps!
Anyway, that is my recommendation. Remember this is coming from an adoptive mother--and incidently, she is some sort of spokesperson for open adoption. She runs a nonprofit that helps family's get information about open adoption.