Please understand I know what it's like to have a child like this, I've had interaction on the child care end of this but not as the parent. I know how frustrating it was for us. I would ask you to consider this.
If he has a diagnosis from a doctor and this can, in any way, be a diagnosed mental illness he might be able to start school now and go year round. In Oklahoma kids with disabilities that effect more than one area of life skills can go to a special program at age 3. It's designed to help them get ahead of their problems the best they can and do well in school.
Your son was left alone and it severely traumatized him, he may never get over this is what I'm getting at. If this is so then that is something you may have to consider. He may be 30 and still doing this. He might also get to 1st or 2nd grade and suddenly make a ton of friends that are always there during the day and realize he has someone around all the time. He might bond with that idea and situation too.
I'm trying to say I understand how difficult this must be for your son's therapist too. There is a distinct possibility that keeping him home full time and you quitting work would build him up enough to where he'd gain that confidence and ability to be away from you when school starts then there's also that part that wants to rip it off like a bandaid and just make him go cold turkey and he'll eventually deal with it.
I would say he needs to go every day of the week so he'd not be confused and think he has a choice to stay home. It's really harder on the kids that don't go full time. Really, it is. They don't adjust as quickly or completely as the kids that go to child care or preschool as the ones that come every single day. Kids understand "this is the weekend" but it's confusing when every other day or so it's "today is a school day" because they get to stay home other days of the week, why not today too?
But if the trauma he suffered at dad's house was so awful and he's not going to get over it then something else has to be done. He can't function like this and neither can you.
I assume you are single since you don't mention a step dad or boyfriend helping. It might help to have a family member move in with you to provide respite care. Someone that will understand this is a trauma and be understanding of their "job". That they're not there to help out in the common way. They'd need to do some training on how to deal with this behaviors and have patience to see the big picture.
No one is expected to deal with what you're dealing with, please know you have my sympathy because I'd not be able to do it. I know that about myself and I know I'd go in the bathroom and shut the door forcibly and lock it, then I'd probably sit on the toilet for half an hour and bawl like a baby.
There are programs in Oklahoma for respite care, for parents of kids with disabilities and they also take in grandparents raising grandchildren. We get several hundreds of dollars set aside for respite care. We get vouchers and when we leave the kids with someone for an evening out or so I can work or something we simply give them a voucher and they send it in to the state then they get a check a week or so later.
It's handy to have for sure. It gives my husband an evening off when I'm working evenings and he'd have the kids alone for hours. We found out about this program and many others through the local social services office. We went in to apply for food stamps and the nice lady gave us a pamphlet with a lot of programs in it. It was O.A.S.I.S. at that time. It's changed now and "better".
So there is hope, getting help and getting him immersed might make it quicker but harder for sure. But his therapist should be getting some help from someone that has more specialized treatment history so you and your son can move the right way even if it is more painful.