Newborn to age 3 is super demanding on a marriage. My perception: most marriages fail when there is a child under the age of 5 in the household. An infant's demands, the baby chores, demands on your time, overall sleep deprivation, and additional financial costs (oi vey, just the cost from diapers and formula alone) will make even more tension in your marriage and distance you from strengthening your relationship. Hey, you've already gone through it; I know I'm preaching to the choir here...
I will be honest... My husband and I have a solid marriage, but after my daughter was born... newborn to 36 months was tough on us. I truly believe that the new baby phase will exaggerate any problems that were in your relationship previously.
If I were in your shoes, I wouldn't tell him 'no' as much as I would say getting pregnant in 18-24 months is a timeline that you are more comfortable with. And that should give you more time to work out the current kinks in your relationship.