J.B.
I think you need to talk to your son's father and then your son and get everyone on the same page.
The matters you're discussing with your son - and that your son and his father are having conversations about - are not the business of a 13 year old. Shame on both of you for the fact that he is even aware of taxes involving him, really. For that, you need to apologize, and it sounds like his father needs to apologize to him too.
Unless your divorce decree or custody/support agreements say otherwise, you are actually not entitled to claim your son as a dependent on your taxes because you do not provide for more than 50% of his support. So if that's a legitimate question in this situation, you were wrong to claim him and if you did claim him, whatever difference it made in your taxes is owed to his father. You're lucky his father didn't claim him and then have you audited. He doesn't owe you any of the deduction he claims for his son if he provides more than 50% of his support. All that said, it's not something your son should even be aware of. It makes your son seem like property.
Regarding holidays...do whatever your parenting plan stipulates. If this isn't spelled out in your parenting plan, it's high time that it is. If your plan states that you have him on Thanksgiving, then you do, end of story.
Your ex is wrong to blame the fact that they didn't travel to see his grandmother on lack on money. If that were an issue, he should have brought it up to you.
At the end of the day, you're both being petty. Sort this out AS ADULTS, follow your divorce decree or whatever was decided by court if you were never married, and jointly apologize to your son and inform him - together - of what the holiday plan is.