I honestly have always had both while breastfeeding. Lately I'm only making a small pot of coffee (between 6 and 8 cups, so only 3-4 mugs of coffee) per day (that is very limited for me since before having kids I drank at least one full pot a day, sometimes more). I also drink in moderation, never more than two drinks in one evening, usually no more than one, and not every night. I don't drink anything until the older kids are in bed or almost in bed and try to avoid nursing but have never done the "pump and dump" thing and do nurse while I'm having a glass of wine (or the gin and tonic I had last night, courtesy of my husband--he's a sweetheart).
Some babies are more sensitive to things I think since only one of my kids was fussy because of caffeine, and with that one I just cut back a bit and he was fine. I just make sure I'm drinking plenty of water (usually 60 to 80 ounces a day) and I think that helps to keep things balanced for me. I try to eat pretty healthy but mostly for my sake since your body pretty much borrows from your reserves to make milk rather than directly from your diet. I feel better if I eat healthy and balanced food and breastfeeding makes me really hungry (especially once I'm working out again).
So pay attention in case your baby is sensitive, but that is not an area I've ever stressed about myself and my boys grew very well. :)