Ohhh we have so been there! We decided when our daughter was 2.5 that it was time for her soother to go. I spent a few days talking to her about it, and explained that if she wanted a big girl tricycle (which was already in the plans, not a bribe, just a matter of timing and reward) she had to throw away her soother. Then we went around and collected them all, threw them in the garbage, took the garbage to the curb and went to buy her tricycle.
We had about a week of rough nights, but she was so proud of herself for throwing it away and getting something "big girl" she quickly got over it. I spent a week sleep training her, I stayed in her room the first few nights, slowly moving to the door. My theory was the soother was her comfort, since I removed one comfort I'd give her one and train her to understand that she didn't need it to be safe. After the first two nights the crying was only a request and then she'd settle, by the 4th night I was in the hall promising not to leave until she was asleep and I didn't. By 1 week she was there. She also had a stuffed animal that was her buddy from then on and over a year later is still her sleep pal.
So my suggestion is give him an incentive to throw away any of the soothers left at home. Once he does, give him the reward, again I don't believe it's a bribe but a reward for doing something emotionally difficult for him. Then if you need to sleep train him. He's 3.5 and big enough to understand when you talk to him about the soother, use things like "you such a big boy now! I can't believe you're so big that you don't need a soother and you can (fill in the blank - clear the table, dress yourself, whatever it is he's proud of), it's ok to be sad you don't have a soother but it means that you're growing up into a man like Daddy etc. I personally always caution against saying, "your not a baby any more" because it's ok for them to want to be your baby and being a baby isn't a bad thing, we just put focus on how awesome it is to get bigger and all the great things you can do when you're bigger.
I wish I had a super human answer for you, but I don't. We all wing it with our kids, doing our very best. Just remember you'll figure this out and he won't be sucking it at 16 and no matter what he'll learn to sleep and he'll always love you.
Not that this isn't long enough, but one more thing. I'd really be careful not to give the soother back to him. It will only show him you don't mean what you say and that crying, even if it's from sadness not temper can get mommy to give in. You don't have to be mean but be firm and mean what you say :)
Good luck let us know how it goes