If he doesn't have a visitation order set up throught the courts, you mom can legally say no to him. That doesn't make it right, but there has to be some kind of formal vistation agreement. He should go to court to get one. If there was a formal visitation agreement, he can take her to court for contempt.