J.A.
Seems a bit soon to take him to the ER, but as mom you are the best judge. If he has been vomitting and not being able to keep fluids down or seems dehydrated, then yes maybe. You said his fever hasn't broken completely...but has it gone down at all? It is really hard to see your child sick.
Does your dr. or hospital have an advice nurse? Mine has one available 24 hours and it is wonderful...they can usually be a good judge if it is time to take to hospital. They can also give you educated advice as what to do. Or is their an urgent care that you can take him to? I'd try talking to a nurse before taking him to the ER.
Be sure to keep him hydrated. He may not want to eat...that is ok, but fluids are important. Let him lay around and do nothing. Virus' do take some time to work through the body...the fever means the body is doing its job.