Your pediatrician is wrong. All current medical/nutritional advise is that children get WHOLE milk and whole milk dairy products like cheese and yogurt. S/he needs to read up on the literature:( They need the fat for brain growth.
"Best Milk for Toddlers
For toddlers without food allergies and who aren't overly picky eaters, they can begin drinking whole cow's milk once they are twelve months old. Keep in mind that the American Academy of Pediatrics recommends that you not give your toddler low fat milk until they are two years old.
A toddler formula is a good alternative if your toddler isn't able to drink whole milk, since they are available in soy formulations, such as Isomil 2 and Enfamil Next Step Soy. Since they are iron fortified, they may also be a good choice if your toddler is a very picky eater.
Although not recommended for infants as a substitute for an iron fortified infant formula, you could also give your toddler pasteurized goat's milk if you wanted to. Talk to your Pediatrician first if your toddler has a milk or soy allergy though."
You're pediatrician should be following the AAP guidelines and s/he isn't. That can be detrimental to your child's brain growth:(
If your child doesn't have trouble with milk-based formula, I'm surprised there is an issue with straight milk. I also haven't heard of a fever being associated with milk intolerance. At her age, it's most likely that it's a cow's-milk protein intolerance, not a lactose intolerance which means she'll grow out of it in the next year or so. It it were an actual allergy, she wouldn't be able to have a milk-based formula.
We're the only species that weans from our own milk (or fake version of it) to another species'. It's really not natural and understandable that so many children, and actually whole races of people, have trouble with it. You can try other animals' milk if you want. Buffalo and goat milk are the closest to human milk. Or, you can use other "milks". Almond is very high in natural calcium and good fats. Soy is quite over-processed, but also has most of what she needs from cow milk. Rice milk is an easy alternative to use for things like cereal, but it doesn't have the nutrients she needs.
When a breastfed infant has trouble with the milk his/her mother is digesting, it is suggested that she stop all dairy products for 3-4 weeks before a difference can be seen. So, if your daughter is having trouble with milk, it will take MUCH longer than just one day back on formula to change it. Many kids can have processed dairy (cheese, yogurt) but not straight dairy (milk, ice cream). So if she's been eating cheese and yogurt and never had trouble, don't cut those out. Do make sure you're giving her whole-milk cheese and yogurt though. It's important she gets the "good" fats.
Good Luck:)