OMGsh! My daughter had the same exact issue!!! She just recently (within the passed month) has begun to use public toilets. You know, I never really had a concern with her fear of public toilets because: 1. they're gross and I didn't want her sitting on one and touching it everywhere (you know how they hold on for dear life), and 2. I didn't want to push it any further than it already was. Plus, I can remember being afraid of falling in when I was a little girl too, so I could relate to her fear. So, we would leave the house after going potty (or at least trying), with panties on. I packed Pull-Ups in the diaper bag (I still carry one since I have a 1 yo), and whenever she needed to go potty out in public, we would go to the bathroom and I would un-stick the sides and put on a Pull-Up for her to pee and then I'd remove it, and put her panties back on. Sounds like a lot of trouble, but it's actually really easy, and she got to the point where she was pretty good at controlling her bladder and would prefer to just wait until we got home or to a private home (if we were out visiting friends/family). Well, one time we were out in public, she had to go potty, and once we were in the stall, I opened the diaper bag to find it empty of Pull-Ups! She had NO option but to go in the big toilet--and she did! I was so proud of her! She got praise like there was no tomorrow. It made me think that perhaps I should have "run out" of Pull-Ups earlier...
We originally used one of those fold up seats (I got mine from Babies R Us) and my daughter did not like it at all. It never fit quite right and would slip on the toilet seats. It also would bow under her weight (she's an average 3yo in size). She did not feel secure and would hold onto me with dear life. Even I felt nervous about putting her on it, thinking it was just going to bust apart and she'd fall in. The last time we used it she got her finger pinched in-between the two seats. I despised that fold-up seat--it was nothing but a nuisance, and just made her fear even worse.
My daughter doesn't like the loud flushing either and will cover her ears when I flush after she finishes. Kids are too funny...