This is one of those questions only you can really answer. But here are some things to consider...
Are you absolutely miserable in your current home or is it simply just annoying that it's not as big as you'd like?
How much of a sacrifice are you truly willing to make? Does it entail going out less and not being able to be fairly "carefree" in your spending or will it be so tight you can barely make ends meet because of our house payment?
How much do you have for a down payment and/or what can you get out of your current home (I think overall, the housing market is in a slump but it doesn't necessarily mean it is where you live)? If homes are selling slowly in your area, you may want to consider trying to save as much as possible for a bigger down payment and maybe home prices will continue to drop (but that means your current house value may drop, too).
Is it simply the fact that you can't eat dinner together? I think your kids are so young, that they probably don't mind sharing rooms.
Can you really afford a house that's big enough to grow into or would you just be buying something as an "interim" home?
We moved a year ago into a house that is bigger than our old house. The rooms may be bigger, but the kids are still always in the family room and kitchen. It really hasn't changed much other than just having more room for our things. But the house we bought is the house we plan on staying in and we had a sizeable downpayment from the equity in our other home.
Anyway, there's lots of things to consider and I'm sure I just touched on a very small amount of them. Good luck in whatever you decide.