O.O.
I think it's going g to be worse since his dad visits only 4x per year.
Why is that?
Is he geographically very far away? (That stinks!)
He's going to have to process why this other child gets to live with daddy every day and he gets 4 visits per year.
I'd definitely look for a counselor -- someone neutral he can talk to.
And be very open & honest with him & his concerns.
I've seen various kids go through this and honesty about feelings is key.
All the best!