My question involves a child custody case from the State of: NJ

I'm having mixed feelings about my child's lack of visitation with his father. I have sole physical custody and his dad has only seen him a few times over the past 2 years. My son is 5 and over the past 2 months, he has been asking about his father. Part of me thinks that if his father wanted to see him, he would go through the court and ask for it. Is it better to have no contact at all or visits with his father very inconsistently? In the past, we couldnt agree on support or visitation, so I filed for custody and support. He was over an hour late for the hearing so everything happened without him there, so he must petition the court for visitation. So right now, everything is on my terms which he has a problem with. I have had many issues with his gf. She has a problem everytime he and I speak. She has threatened me with calling the police if I have their father in my home, having him tested for drugs and getting my kids taken away. She even texted me that she has a contagious, infectious disease?? She is obviously mentally unstable and has been arrested for drugs and has DYFS involved with her own children. So, I don't want my children around her at this point....after all, it's about his father bonding with my son, isn't it? He feels that I'm "controlling" everything. I can't stand that he is being that stubborn and my son is feeling it. I'm getting very angry. So....should I just let it be? or reach out to his father to see if we can work something out because our child is asking for him?