my son did the same exact thing
he is finally sleeping through the night at FIVE!, and that is because i am pregnant and got extra tough with him.
he has a very strong personality and such a loud screamy tantrum that we would give in to him for an easy life, plus in the middle of the night its easy to give in when he wanted to come into our bed! it would not have been so bad if he wasnt the world greatest fidgeter, he would end up sleeping horizontally accross us whilst twirling and kicking (in his slepp)
we would have to lie with him, sometimes for an hour till he fell asleep, otherwise he would cry, say he was scared, thristy, hungry - whatever!
so this went on for 5 years - and unless you get tough yours sounds like he will do the same - this is what i did and it may shock some, but its old fasioned and it works in just a few days.
my friend who is jewish gave me a book on disciplining children, after watching my son who is bossy and has a low frustration tolerance, and i read it.
they advocate switching - not with a massive stick or anything, but a paint stirrer from home depot, they make a loud noise, but dont really hurt (i tried it on myself first)
you explain to your son whats going to happen - he has to stay in his own bed, or he will get the switch.
my son got out of be ONCE only, i switched him, he screamed for 10 minutes, i told him if he didnt stop he would get another - he stopped.
in the middle of the night when he crawled into our bed - i did the same thing - but i felt a little sorry for him, so i have given him an option.
i made him a "special" bed on the floor in our room so if he is lonely or scared he can lie in this special bed which i deliberately have not made too comfy, its just a ready bed on the floor.
he very rarely has slept in it, when he sneaks into our bed i tell him, no boys in mam and dads bed, go to your bed or sleep in your special bed, if he doesnt get out i tell him i will get the switch, and he goes.
i have not had to actually use it more than a handful of times, but it works great on all types of behaviours.
i will try and find the name of the book for you to read.
sorry if this is long winded, but i dont want you to go on like i did for years.