There are so many other reasons to go to court. You really have to protect yourself, your child, and your ex by having a schedule and payment agreement that is memorialized in court filings. If either of you decides to be more generous with the parenting schedule or money from time to time that's fine but when things go sour, you both need to be able to fall back on more than your verbal agreement.
With a child support agreement in place, he would basically be paying you directly (through wage garnishment) for the expenses like camps and after-school care that he is covering now. You would use the money to pay for those big childcare expenses but it's also there to reflect that having a child live with you increases your cost of living (one more bedroom, higher utility costs, school expenses, groceries, clothing, etc.) and offset those costs that he doesn't have to absorb.
You don't even need an attorney for this...if you apply through the DOR's child support enforcement division, they handle notifying him, provide paperwork to both of you, schedule the court date, and send an attorney. During this process they will also formally establish custody and can enter a visitation record (either a default recommendation or something that you two agree on in advance ) into the order. Know that the attorney represents the DOR's interests and not yours per se, but as the custodial parent, their interests and yours align. Once the order is granted, they will start wage garnishment and you will receive a debit card that they load up with his payments whenever they get processed (which is based on his pay schedule). This way, the payments get made without either one of you having to do anything or see each other.