This is a hard question to answer because so much important information is missing. I would need to understand the context of the situation to offer good suggestions because different situations lead to very different answers to your question.
A lot depends on how the boy’s relationship with his father was while the two of you were still together. Did your son and his father have a positive, loving bond in those first 4 years? Was his father around and emotionally available to him during those first formative years? And how did his Dad react to the divorce?
Regardless of the reason your relationship broke up, what is key to a child’s future relationship with the non-custodial parent is how the two parents handle the situation. If they keep their adult feelings of anger and frustration to themselves and not verbalize a lot of those feelings in front of their children, their children are more likely to maintain their own independent relationships with both parents without excessive guilt or conflict about loyalty. However, if you put the child in the middle and constantly badmouth the other parent, this can certainly create discomfort and emotional turmoil for the child about the parent with whom they do not live.
Of course, you also need to find out if there are specific negative or inappropriate things happening to your son when he is with his father. Being around a parent with active alcoholism or drug addiction, a parent who is ignoring their child during visits or subjecting him to verbal abuse, and of course being physically abusive or sexually inappropriate toward the child can all lead to feelings such as those your son is experiencing. If his feelings are occurring for any of these reasons those issues need to be taken very seriously.
First, it is extremely important to rule out any potential abuse or neglect that may be occurring when your son is with his father. If any of these things are occurring, it is vital that you take whatever actions are necessary to ensure the safety and well-being of your son, regardless of the other parent’s wishes.
If none of these issues are present, it is important to take the issue up directly with his father and see if you can find out what things look like from Dad’s perspective. If none of these approaches yield satisfying answers, I would suggest taking your child to a qualified child psychologist or counselor and perhaps even consider counseling for the family as a whole, whether or not you are in an active relationship with his father. Even parents who are no longer in a marital relationship may need the benefit of an outside professional perspective to untangle the reasons for your son’s feelings and how they should be handled.