You're either in compliance with the court order or you are not. If you are, his threat is meaningless.

If the issue is that he's trying to change a previously agreed visitation schedule, and you can document his approval of your travel schedule, on its face it would seem silly of him to try to convince a court that you somehow harmed him by planning out-of-state travel based upon his consent to your travel dates.