You should try to locate a doctor who specializes in sleeping issues for children, if possible. One thing I learned the hard way, but it is the NUMBER ONE mantra: Every Child Is Different. That means a lot of things, and one of the things it means is: every child will respond differently to a particular solution or treatment.
I have three boys, ages 7, 4 and 1.5. My first boy -- my 7 yr old -- had significant sleeping difficulties punctuated throughout his life. The other two boys don't seem to have the problem.
It began when my 7 year old was about 6-9 months, I believe. He would not just wake up crying or screaming or something... He would wake up FRUSTRATED. I knew it was his "frustration voice" because I got used to what it sounded like. But what was weirder is that when he started fussing in the middle of the night, he'd often sit up and fuss while staring into space. He wouldn't look at us, wouldn't acknowledge that we were there, nothing. The episodes got worse and worse. When he was around 2-ish it really got bad.
One of the episodes I remember best was the "Harold the Helicopter" night. Harold was a little toy from the Thomas the Tank Engine collection, and he loved his Harold. He would always hold it tucked between his thumb and index finger and walk around with it much of the day. He would have huge fits if he didn't know where his Harold was, so it became sort of a security blanket. So we let him sleep with Harold and one night all hell broke loose. He sat up making his super-frustrated grunting and whining sounds, which slowly devolved into crying, and that spiraled into a complete non-stop breakdown that lasted an hour, at 2 am. My wife and I tried everything to stop him, but it was like we were invisible. And what he was doing was this: He'd hold his Harold in the air and drop him onto the mattress. Then he'd have a fit and try again, and again, and again.
Episodes like this would persist nightly for 2-4 months, and then stop... Suddenly he'd sleep through the night like any other boy, and we'd think the problem has passed. Then after a couple of months it'd kick right back in again for another 2-4 months.
They were not "night terrors" as is commonly thought, but I now know they are related. He was having some sort of delta sleep parasomnia. Sleep walking, teeth grinding, bed wetting, night terrors (not nightmares), and restless leg syndrome are all delta sleep parasomnias. Delta sleep is the 3rd and 4th level of sleep, the deepest level. You don't dream in that level of sleep, at least not in the usual way. But parasomnia will cause very strange things to happen, and the clue is when a child seems awake -- eyes open, moving around, making noise -- but doesn't seem to acknowledge you're even there.
I don't know if your child is acknowledging that you are there or not, but the reason I am stating this is because we tried a common doctor's advice: let him cry for X minutes and then sleep with him until he falls asleep; next time let him cry for X+5 minutes and then sleep with him; and so forth, each time increasing the X minutes. Unfortunately for our poor boy, this is what made his problem worse. I believe his parasomnia started due to a separation anxiety that began when we were attempting this method when he was 6-9 months old. It terrified him deeply to be alone, and ever since then it adversely affected his delta wave sleep and he developed a form of NREM parasomnia (NREM meaning Non-Rapid-Eye-Movement; as opposed to a REM or Rapid-Eye-Movement parasomnia, which are different is they occur while you are dreaming).
The lesson learned: methods like this that may have worked with other children actually gave our child problems and made them worse. Every child is different.
Our boy's problem always went away immediately when he slept in our bed at night. We would let him do that for a few nights and then try letting him sleep beside our bed on an air mattress and finally move him into his room, but it would start all over again.
In most countries, excepting the US and possibly the UK, babies and kids sleep with their parents for the first several years of life. I learned this while traveling, especially as my wife is Taiwanese and we spent a little time in Taiwan. The US is one of the few cultures who thinks it is better for babies and toddlers to sleep in a room of their own right out of the womb, and other cultures are actually horrified at this.
While I can't judge the right-or-wrongess of the US cultural notion, I can tell you that our kids always felt better sleeping next to their mommy and daddy, and there has to be an instinctive and deep-rooted meaning to that. Some kids can sleep by themselves pretty quickly -- our second boy was especially easy. But every child is different, and some kids experience a lot more separation anxiety. (We could not give our first boy any seclusive form of timeout. It was traumatic for him and not properly disciplinary. So "timeout" is another highly-preached methodology these days that I would only accept on a case-by-case basis.)
After learning this the hard way, we decided to just deal with it... We let him sleep in our bed when he needed to. It took a lot of patience. When he got to about 5, he started feeling more and more comfortable about sleeping in his room, and sometimes in the middle of the night he would get out of our bed voluntarily and go sleep by himself!. During his 6th year he hardly slept with us at all -- only a dozen times at best the whole year, and not all night.
His parasomnia has turned into a sort of insomnia, where he wakes up at 1 am and cannot go back to sleep. So what I have done is occupy him. I have a speaker with a wireless connection to my iTunes that plays audiobooks on a loop, and he'll listen to that sometimes and go back to sleep. But my wife hates it and thinks he will develop a dependence on it. I tend to disagree, I think it's a sleeping aid. But the alternative is better anyway: I set up a reading lamp over his bead with a cool-burning florescent bulb and he'll flick that on in the middle of the night and read a book. That makes his eyes heavy and he'll drop back off again, and it also beefs up his reading skills. Win-win.
I would advise trying to determine whether he is waking up during REM or NREM sleep, which might require the help of a doctor who specializes in sleeping disorders (even if he may not have a disorder), and also just being patient and sleeping with him the way he needs it. His "tantrums" may be interpreted as such, but it may be a mixture of anxiety (insecurity) and simply being overtired, as kids his age still get very cranky when they are tired. I don't think he's just trying to get what he wants, like a particular food or toy, as in a real tantrum. He needs the comfort and security that only you offer him to sleep through the night, and you need to provide it for as long as possible. It won't last forever. At some point he'll develop enough confidence to sleep by himself through the night.