You cannot use visitation as a blunt instrument like that but you can enroll your son in activities that the dad will either take him too or not be allowed to pick him up every other weekend. How much time does he spend with your (plural) son?

Also, I suggest you go to court and get his tax returns intercepted and threaten his driver's license be taken away. There are things you can do to force compliance.