I think go by what your Ped said. A kid's milk intake can affect where he/she is at on the growth chart. For example, my son is very petite and at 6 he should still have milk with every meal. It has more calories than juice or water. Of course he also eats a meal with it. He still likes to drink a glass of milk before bedtime. So... if I look at how much milk in a day he takes it's probably actually more than your child is taking in with his meals and snacks and before bed.
However, my friend's daughter is sturdy and high on the growth chart. She is 3 and her ped said only one glass of milk per day. Then water.
So, like I said, if your ped says you're doing it right, then don't worry about it. After all, in the above example by their ages you would assume the younger one would have a higher milk intake. Milk is important in those first few years to prevent rickets and other bone diseases, so I wouldn't worry about it.