First, brace yourself. What you fear is likely accurate. I don't mean you can't survive, I mean you need to be very strong for your son. You note the father has not shown care or concern for his son for several years, so I imagine there is a lack of an emotional attachment between the father and his young son. This is generally required by state law to excercise full equal rghts to parent a child. If the father left the state, then I would infer he has performed minimally to no parenting functions in the course of his child's life. This is also generally a legal requirement that could restrict time or decision making. You'll want to document your timeline and how your son got care. Be sure the father cannot show that you have tried to restrict his care of his son, and you want to be sure you have encouraged this realationship between father and son. It would be best if you had documented evidence of his drug use, prior crimial behavior, job stability, time he actaully spent with his son, occurrances, length of, and period between and calendar time he maintained the relationship. If you see a lawyer right away, you'll need money. The more you have the easier it will be for you to have the final outcome be what you want.
Family court proclaims to be interested in the best interest of the children, but I have found this to be mostly an empty mission. Family court's apparent real objective is to perpetuate their existence and even increase the number of necessary paid personnel including lawyers, guardian ad litems, parent evaluators, mediators, facilitators, parent helpers, paralegals, and Judges as well as support staff to keep the machine.
It is important to consider the Order in place covering the residental schedule. It sounds like there may not be one that allows your son to spend extended time with his father out of the state. If it doesn't and he petitions for this change, unless you have documented evidence that he is not able to care for his son then he will likely get it.
Bottom line: I suggest you figure out what you think would be acceptable in your view as this child's mother. This is the first step. If the answer is one night every other week, fine. If its 2 weeks, fine. YOU KNOW BEST.
Once you have this, and don't let anyone tell you otherwise, you can interview attorneys to find one that believes you know what is best for your child as the only expert in this childs life. If the attorney is not on board with what you know is best, seek another consult. The choice of attorney is the single most important factor in this. Pick someone that tells you to your face, and you believe, that he/she agrees with your suggested residential time and he/she will do anything possible to get the Judge to agree it is in the best interest of your son to create a schedule as you suggest that will support what is in the best intereset of this child. Make sure the attorney also believes that you absolutely do support your son's relationship with his father, and have no ill will or desire for any form of retaliation over your past relationship with the father. You only want what is best for your son as the primary residential parent and only one for most of the child's life To do this you suggest keeping the child in the stable, consistent environment in which he has thrived. Extended residential time could be considered if the relationship, previously non existent, develops into something you feel comfortable is good for your son.
i am not sure, but I suspect your ex husband's desire to be with his son is related to his financial responsibility. Just a hunch. This is fine, and he has a right to parent his son. BUT, be sure you are clear that your number one priority is what is in the best interest of you child, and it is the only thing that matters, not money, not your time with him. You want your son to have what he needs to grow into a self confident, decent man who can make informed choices about life and that includes knowing what a healthy relationship is and the boundaries of such.
I also think what is often understated: a child doesn't have to live with someone to love them deeply or to know someone loves them. The child knows this by how they feel when the person calls, whether the person seems to care about what they are doing every day, if the person asks what they might need, or desire, and then following up to help or find out how things are going, the child will feel loved and ca form a strong relationship with the person, father or interested adult.
So, if you determine that your husband is going to file, find an attorney quick. What I suggest is you make the search for an attorney a high priority and if needed take time off work to find the person who truly believes YOU know what is best for your son and will commit to fighting for this result. I am positive that you know what is best for your son, but no one in family court cares about what you think and no one in family court really cares about your son or you or the father - please believe me. They only care about how they can make money or maintian power and control. So, its crucial to have a legal advocate that you trust to act on your behalf. This may seem overstated, and you may think any attorney you hire is supposed to act on you behlaf, but let me tell you from experience, there are many attorneys, seemingly educated and appearing to care and want to represent you, as long as you have money, but they are just going through the motions (no pun intended) and if things get sticky, or seem more difficult than doing nothing, they won't have your back. You must have someone you instinctually know based on multiple interviews, is going to back you up and fight for your right to parent your son because he is completly in line with your suggestions for what is in the best interest of your child. Without this, you are in for a long traumatic ordeal, and protecting a 4 year old from it will not be possible.
Last, the power is to keep supporting your son to trust himself, firmly know you love him, you value everything he is inside and out, and you want him to be in a stable and safe environement so he can grow up to make good choices that support his values. If you do that, no matter what happens, and things may be painful, he will eventually speak for himself and then your job gets easier and you can just support his beliefs, support him to speak so his voice is heard, that is how he reclaims his rights, his right to have consistent love and care regardless of who he is biologically related. No one can deny his voice forever, no one will be able to ignore him once he is aware of his own power to decide his fate. You can help him get there. Never deny his gut feelings, or yours. They are right and they will keep you connected to him always. I'm sorry this is happening.