You're fine! People will ALWAYS find a way to criticize you for something, but don't worry. There is absolutely nothing wrong with taking a 5 week old out.
When they are super little, just keep them close to you in a baby carrier or safely in their stroller/car seat so they aren't getting too many germs and there's no problem at all. Being out on a walk in the fresh air is great. It is unrealistic to keep a newborn at home when you have older kids.
My second was out pretty early too. Don't remember exactly, but my son was in school so it couldn't have been too long since we had to take him and pick him up!
When my oldest was a couple of weeks old, I went to a lactation center for help. Afterwards, we stopped to pick up food at a fast food place next door. I sat with him in the very back corner - the restaurant only had a handful of customers, none were close to me - while my husband stood at the counter to order. An older woman, who spoke very little English, made a point of coming TO MY TABLE to tell me how terrible it was that I was out (in her culture, you are supposed to stay in bed for a full month - not just the house, actually in bed with the baby). I told her my baby was coming back from the doctor and we had no choice to be out. She continued to tell me how awful it was until I finally said something like "you are the one coming up to him and giving him germs."