C.D.
I have a bird nest in the gutter over my bedroom window and just don't have the heart to take it out when there are eggs or babies in there. The little baby bird are so sweet! These birds aren't scaring my kids, so I don't have that to worry about. That's a tough one!
Whenever you are at a point where you need the bird to go away, my Dad told me an old trick that sounds silly, but they say it works. Get a realistic looking rubber snake and put it somewhere close to the nest. The bird can't tell if it is real or not and will take it as a threat and stay away. Then you can safely remove the nest and the bird will make a new one somewhere away from the "snake".
Good luck!