I experienced the exact same thoughts as you...what happens when the time comes that my daughter has contact with him at an age where I can no longer make decisions for her and he lies to her and tells her I kept her away from him?
So I asked her what she wanted to do and explained that he had let her down (due to his illness - alcholism) before so there could be a chance that he could do it again - however we were going to give him a chance, so she could make up her own mind up which is what she wanted.
He came back into her life, I allowed supervised visits over a period of about 2 months with myself until unfortunately he let her down AGAIN...she dealt with it all very well and said to me mum I don't want to see him, he's not like us and that was 1 year ago. I have always spoken honestly and openly with my daughter about her father (without bad mouthing him yet being honest) and she eventually decided she didn't want anything to do with him, this was her choice, he has since sent her a christmas card, and missed one birthday but she looks at them and puts them to one side without it being a big deal, I do try and talk to her to see if it bothering her as I would'nt want her to feel bad inside and not say anything but she really is not bothered, he himself has done that.
I had allowed her the freedom to see him if she wanted, being 100 percent honest with her and like you, didn't want my daughter whom I have worked hard raising and supporting to grow up and be torn between lies that he may tell her, now she is fully aware of what kind of person he is and doesn't trust him, this isn't what I wanted for my daughter, but as my daughter said at the time 'but mum, he could of changed'.
He let her down, however she saw first hand because she wanted too, (I had asked her previously if she wanted to meet her dad and had told her the reasons why I had kept her away from him) but I felt she was now of an age where she could understand and make her own decisions, the only thing that stood out was the fact that I had done my job of raising her well, as she wasn't prepared to be treated like he wanted to treat her as she had never been treated that way and it didn't effect her as he had never been a big part of her life so therefore he wasn't important, Ii could of been that he had turned his life around and treated her well (which he had the chance too) but unfortunately he hadn't.
I explained to my daughter that being a father is about caring, supporting and wanting the best for your child and that is what makes a 'father' not a person who carries the title 'father' and as my daughter has many people who care very much about her and support her in all she does, she knows the difference.
So I would just be honest, open and ask her your daughter if she has ever wanted to see her biological father and perhaps explain that the outcome of seeing her father should it happen may or maynot be a happy one and that you stopped her seeing her father as you felt he wasn't treating her right and you want the best for her, nothing can take away the fact that you and your husband have been the ones raising and supporting your daughter and I know first hand how scary that is to think that someone could come and unsettle that, but trust in yourself and the fact that you have raised your daughter and that she can handle this and so can you, then like me you can get on with your life and can either engage in a relationship with your daughters father for the better or forget about this man who calls himself a 'father' if he has empty promises.
I too, had grandparents on his side arguing that they wanted to see their grandaughter, but having given them a chance (and not the first time) they didn't act appropriately - once I said they could see her for an hour, in this time I was away I asked that she didn't play in the cul-de-sac as I didn't allow her to play on the street, they then proceeded to ask my daughter to lie to me and say it was ok because 'nanny' said she could (baring in mind, this was a woman who hadn't seen her grandaugter in years and assured me that she wouldn't allow my daughter to play on the street) my daughter rang me on her mobile and told me, I collected her and since then they haven't seen her again, I gave them the chance and they didn't repsect my wishes.
Each case is different and I listened to many people telling me, you have no right in stopping her see her father/grandparents,she will want to see them and you will be blamed for stopping her, in the end I went against my mothers views and chose to do what I did, but it turned out for the best as we both of learnt from it and now my daughter and I can get on with our lives - but only you can decide what is best for you and your family.
Good Luck in whatever you decide. ;0)