Just email everyone and say "we're just having a small party this year." Really your son won't remember his bday anyway. My son will be two years in a couple of weeks and we're skipping his bday party this year. Everyone who came last year are really busy this year, so I don't want to stress everyone out, or hae noone show up. Yes presents are nice, but he has plenty of toys and clothes, so we don't really need it.
Small party - you have great suggestions already. As far as themes, I was going to do a lolipop theme (stick lolipops in the cake, then serve food on sticks - like kabobs for adults, corn dogs for kids). My son really likes lolipops, so the "food on a stick" sounded fun. My son also loves M&Ms, so was going to maybe instead do primary colors, have bowls of M&Ms out, dress us all up in primary solid colors, maybe buy him an M&M shirt, then make cupcakes with M&Ms all over it.
Mini sandwiches are great idea. You could cookie-cutter shapes out of regular loaf bread sandwiches, the cookie cutter going with your theme you choose. Someone else a while back suggested 'dips' .... do queso, do guacamole, do veggies with ranch, etc. Just lots of bowls with lots of dipping options.
Good luck. Have fun. Remember that your son won't remember any of it, but pictures are always nice.