Let everyone go outside, parents supervise their own kids (you don't). Just say "help yourselves to the swing set" and be done with it. If a child asks if he can play there, say, "Yes if your mom or dad says it's okay and will watch you." Nothing lame about it. You don't have to schedule and supervise and organize everything.
Don't decorate outside other than perhaps a balloon on the mailbox to identify the house for anyone who hasn't been there before. If you can borrow a croquet set or a couple of frisbees or possibly a cheap yard game, go ahead. But don't do a lot of set-up and just everyone play as they wish - you'd be surprised at how resilient people are. In fact, they will appreciate that you gave them choices and didn't expect everyone to sit in the house the whole time. I disagree that you have to be outside the whole time. Periodically, you can scoop up your baby and do a "lap" outside just to say hi, and if you want to put a cooler of drinks out there ahead of time, you can. Believe it or not, if you have 12 year olds, someone needs to supervise things like beer though! So keep that inside and/or ask a trusted friend to be a second set of eyes for anything that's needed.
The ice cream truck is more than 2 seconds of fun - they will love hearing the chimes, looking over the menu and talking about something that so few kids get to do anymore.
You can go to the craft store ahead of time and get some foam shapes, googly eyes, glue sticks, etc. But just put them on the table and let an adult help supervise. This is also a good job for your 12 year old, to organize the crafts and supervise the little ones using glue. Give her $10 if you have to but make it her job.
Get past the viewpoint that you have to be everywhere and do everything. I am the type of hostess who greets people, shows them where stuff is, and let's them know to help themselves. If they go home hungry or thirsty, it's not my fault! I do a lot of make-ahead recipes with no more than 4-5 ingredients each, and that's it. I pre-label my platters and serving pieces with a sticky note so that I know what's going where, and then I just serve it all up 15 minutes before guests arrive - no digging for plates and silverware. I put out a can labeled trash and another labeled recycling, and hope that guests get the hint to help out.
But scale back your ideas of what a "hostess" is supposed to do! You have 3 kids and a household, and the point of this party is to celebrate the 1 year old's birthday, not for you to be Martha Stewart. If you have guests with higher expectations, you need to re-think your friends and who you invite to family-friendly, kid-friendly events. But my guess is that everyone will enjoy a laid back, not overly scheduled, casual party with food and fun!