A.V.
Personally, I'd pick milk first and then see if it's just dairy or something else. The problem with cutting out both at once is that you don't know which thing it was. Then I'd go back to milk if that wasn't it, and cut out one veg (like broccoli), etc.
You might also call a La Leache League leader or certified Lactation Consultant to get her take on it, too.