For us, 1 to 2 was much harder than 2 to 3 was. I take my 3 boys everywhere, almost always alone, because my husband's job doesn't allow him to join us for 99% of the things we do. Yes, it can be quite chaotic, but it is also very happy. I love seeing my boys growing up together, and we'll be adding a 4th to the mix in September. I am also 31, but I don't seem to feel my clock ticking. I am the 3rd of 5 kids in my family, and my mom had me when she was 31. Plenty of time left to make a decision.
Our family size is open ended. We are ultimately leaving it up to God, but after #4 we will not be trying to conceive again any time soon. I plan to lose a lot of weight, and to get our house in order to sell/move. Hopefully we'll be able to have a 5th in another 3-4 years, but if it doesn't happen, I do already feel like we have a complete family.
Every family is different because we all have different capabilities, tolerances, support structures, etc. Although I am not typically a person to speak in relative terms, when it comes to the size of a family, it really is relative. What works for my family wouldn't work for my sister's, or my neighbor's, or my best friend's.
ETA: In my family of 5, there were no odd men out. I certainly never felt that way, even as a middle child. With my 3 boys, there's no odd man out, especially since it usually is just me + 3 when we go places, so we have our even # there. When my husband is with us it is even better, even though we then have an odd #.