While I believe this is a very personal decision, I think it's an excellent question. There are a lot of personal decisions like this: some people care a huge amount, others happily accept whatever happens, and of course there are many who fall somewhere between those two.
I think it's excellent that you're husband and you both feel the same way, even if it's 'don't know' at present.
I decided to stick with two for a couple of reasons. I'll make a list:
1. I only have two hands, and I couldn't imagine what I'd do if I was holding two and one went another way. I just couldn't think what I would do.
2. I watched a friend nearly lose her mind, with three kids each 3 years apart -- the kids are at such amazingly different stages at 6, 3 and newborn. She said it was like being pulled in three directions at once... for nearly three years.
3. There are twins in my family --the kind that run in families.
4. We couldn't keep our small car, our house would become tremendously cramped (rather than the 'cramped' it already was, with 1100') and the 'expensive time' I could see coming -- sports, arts, activities, travel, would be appreciably more expensive.
The last one, I was aware of, and really was the last in the list of importance. We could certainly handle it, it would just mean staying in the same house a lot longer, having a very old car a lot longer, having very little disposable income for a lot longer...
Once my youngest was out of diapers, the question wasn't so much 'do I want another baby in the house' because I adore babies, loved being pregnant and would happily have had another 4 if someone else was going to do all the other housework in the building... and the economics were more favourable. But the question became 'am I willing to jump backwards 3 years in my life to the routines and realities of babies?'
I already felt extemely stretched to meet the needs of my two little ones according to the standards I wanted to live up to.
Ultimately, it was the satisfaction with our family size (two was good, three would have been fine, if it had happened before we made up our minds) and the lack of desire for another year of intensive baby needs, and a complete willingness just to enjoy these two little people growing and maturing.
Today, I've very happy with that decision. I hope you find a decision you are very happy with, too.