I'm with Lucky here.
Your husband refuses to use condoms? Then he can stay home and look after the baby that will result if he insists on sex with no other birth control.
He probably will complain how "un-spontaneous" things are if you are using the rhythm method and therefore checking your temperature and a notebook constantly at the mention of sex. He wants spontaneity and sex on demand? Condoms. (And even then, condoms are not 100 percent perfectly effective - they can leak and tear.)
He's putting 100 percent of the responsibility on you alone.
The alternative, if you really are ready to stop having kids, is to get your tubes tied. You say you're not ready for that, but also say you don't want more kids. I think that translates to, you're ready but maybe don't want to go through a medical procedure--? Consider life with a fourth child carefully and ask yourself if you really could handle it emotionally, financially, if your marriage would suffer. Then think again about having the procedure -- as well as about why he is putting all the BC responsibility on you.