J.R.
So does he also not think that you should "influence them one way or anther" regarding healthy eating, exercise, visiting the Dr...
As a parent, your job is to influence. You are to shape and build your child into what you believe is a productive and moral adult. As the child matures, the child then has to choose whether to continue on with the same choices the parents made, or choose other choices. Do not be fooled by the idea that not taking your child to church is being "impartial" and "not influencing." It IS influencing--it telling your child that you do not believe that church attendance is important.
I am a conservative Christian, and my heart goes out to you. Let me please urge you though to remember that you are to submit to God first, and your husband second (this is a mutual submission, not a "husband as king of the house," as your husband is also directed to submit to you...but that's a whole 'nother HUGE topic). God tells you in Dueteronomy that you are to be constantly teaching your children about Him, teaching His commandments to them. That *may* include church attendance--you need to pray to God to get clarity on this. If you feel that obediance to God requires you to take your children to church, then you should. Even if your husband says "no," you must submit to God. In the book of Acts God sends the very powerful message in the story of Ananias and Safira that a wife is NOT to submit to her husband to the point of sinning against God.