Have you thought about a second opinion or seeing a pulminologist? My son had a lot of breathing problems when he was a baby, would cough up large amounts of mucos, had been hospitalized for pneumonia, the whole nine yards...I love his pediatrician and understood his delay in diagnosing asthma. At visits he only had a noticible wheeze every once in awhile, and the doctor was reluctant to put him on inhaled steriods with out first having solid proof there was an actual pulimanary issue at hand. Obviously you don't want to do that to anyone without knowing they have a real problem.
When my son Matthew went in for his second set of ear tubes, they were to remove his adnoids as well but could not intibate due to the wheeze he had that day. Two days later we were outside playing ball when I saw how out of breath he was from playing just for a few minutes. I called the doctor and basically told him he was gonna give me a diagnoses od asthma and put him on some medication or I was seeking treatment elsewhere. I gently reminded him that I understood his relutcance, but I'm his mother and I live with kid, this IS NOT right, he's sick!
A couple years later he's on a Singulair tablet and 2 puffs twice a day of Flovent and in relatively good shape. The infections have ceased, and though he still gets winded after rough play occasionally, it's very seldom and he's a much healthier, happier kid.
I wish you the best of luck in finding out what's wrong. I know how hard it is to have a baby that is sick non stop, in and out of doctors offices, and being a sleepless mommy! I would try a second opinion and just see what another doc has to say. Understand the reluctance and how dangerous it is to put a kid on drugs they don't need, but also maintain that your his mom and you know there is something more going on.