I can totally empathize. I have an almost 5 year old that I've co-slept with since he was a baby. For the first years of his life, it was just the two of us, so it was really easy. Now, I'm engaged, so that presents a little problem.
Currently, my son has his own bed in my room, which is sandwiched between my bed, and the wall. His mattress is about 4 inches lower than my mattress, so that keeps him from rolling onto my bed. I still lie with him in his bed until he falls asleep, and he, too, will wake sometimes in the night for either a drink, or from a dream, or just to be comforted back to sleep. I do have to say, though, that he does fall asleep pretty quickly, and when he wakes, he goes back to sleep quickly, too. This actually works pretty well for us. I am in no hurry to push him out of my room because his room doesn't have any windows, and I will not allow him to sleep in there for safety reasons. So for now, our arrangement is the best it can be. Just an fyi, my dad built an apartment for me on his property when I was a young, single adult, and with the lay of the land, the one tiny spare bedroom/office room was built back into a hillside, so it couldn't have a window, which didn't matter at the time. Plus, I do enjoy our snuggle time, and I love him falling asleep close to me. I have no regrets about that, whatsoever. He was a high needs baby, and although before I had him, I swore I'd never put a baby in my bed, when I realized how much better we both slept when we slept together, well, it was a no-brainer. Anyhow, we will be moving into bigger home in the spring, and I plan to start having my son sleep in his own room then.
I have the same worries you do in terms of him feeling pushed out. I plan to have another baby, God willing, and I have felt that he needs to be sleeping on his own prior to that because I can't have him on one side, and the baby on the other, and the husband, well, wherever he can fit. So I plan to try to transition him as soon as we move. I will still make a place for him near me to come in the middle of the night if he needs to, but I will have him at least start out in his room.
Your situation is a little different, but maybe the best solution for you would be to actually move your son into your room, something like I have now with a twin bed next to yours? If you have the mattress height difference like I do, that would keep him in his bed, you could still be with hubby in the big, comfy bed, and have the co-sleeper on the side by daddy. Or, you could put the co-sleeper on your side, and just put the twin bed in your room, but not next to your bed.
I know you will get some grief here for co-sleeping, and you will be told that you created this dilemma by co-sleeping, but don't beat yourself up about it. You did what felt right to you and your son, and only YOU know what that is. Let the rest roll off your back. I do.
Best wishes to you...