When my daughter was in 3rd grade and struggling with math (with massive arguments every single night), I went to speak with her teacher (who I knew quite well). Her teacher said to me, "you know what her biggest problem is, don't you" and I said, "no". Her teacher said, "you". Of course I burst into tears (the teacher was absolutely right!). The way I was handling it (similar to the way you are) was eroding her self-confidence terribly and making the "problem" much worse. I immediately found a tutor and my daugther went once a week to her house. While she initially resisted, I explained to her that having a tutor meant that the time we spent together would be spent doing fun things, not fighting. It was a dream. She has since become a champ at math, getting A's almost always AND, more importantly, her self-confidence is very high in this area. She is in geometry now and has struggled a bit so she is having a tutor for the first time in about 4 years tomorrow evening to help her prep for a test - she is very open to it.
That said, my sister is a neuropsychologist (works with kids who are having learning struggles/differences) and it is about this age that more neurologically-based challenges appear - this could be what's happening with your son. Tap into school resources (they make it very difficult, so beware) as you don't want to mask the problem with a tutor (I have a friend whose daugther got good enough grades as a result of working with a tutor and it led to much, much more serious problems since she was excluded from special services - be careful).
Most importantly, work with your child - not against him. I believe he is doing the best he can. If he can't get, please do not reprimand him or get angry - he needs your help and support - that is our job. I learned this the hard way; now, when I see one of them struggling, I take a deep breath and become an advocate, not an "enemy".
Good luck.