First off- YOU'RE the MOM, he's the doctor! Listen to his advice, but do what feels right to YOU and your son's father.
My son was drinking about a half gallon a day at one point... before he even TURNED 2. I was still BFing him until 18 months, and when the BFing stopped, it took 2-3 9oz cups of milk to get him to sleep... TWICE a DAY! even at just 2, that's still 36oz, BEFORE including meals. My ped said to try to increase the dairy rich foods in exchange for the milk, but not to stress about it at his age. He also advised to "thin" the milk with water for bed/nap and transition him to water only that way. DIDN'T WORK!! I could give him about 90% water, 10% milk, but straight water was a no go! It wasn't until about 27m that he would except 1 cup of milk, followed by water. (He's now 2y 7m) My ped also told me that getting him off a bottle is not as important at this age if you are still bfing. (Don't know if you did/are.)
You might try putting REAL milk in a sippy and plain water in a bottle to see if he's willing to switch. Please, DON'T add any flavor sauces to it, as you will be making your life misserable later with the extra sugar it adds. Another trick that MIGHT work is going back to the 6oz bottles. Many times they keep drinking it BECAUSE they have it, not because they need it.
In the end, you need to be the judge of what's right for your child. The doctor's job is to tell you what "pediatric guidelines" suggest. SUGGEST!!! Not REQUIRE, SUGGEST! BTW~ I've never heard of it causing Anemia, but if so, ask about an iorn suplement.
I hope I've put you at ease. I'm sure you're a great Mom~ J.