R.K.
Knock on their door with a plate of cookies or basket of muffins and 2 coffees, one decalf for the Mom. Simply say "Hi, I just wanted to officially introduce myself and say congrats on the baby". Ask to see the baby's room and just get to know her. Ask if she needs anything and let her know you are available to help. As you know the first couple of weeks with a new baby are difficult, no sleep and horomones raging does not make a pretty package.