If the court ORDER says you need permission from the FATHER and or the COURT, then that is what you need to do. And yes...They can FORCE the return of the child.
Violating this order will place your primary custody in jeopardy. Along with your obvious disdain of the father by referring to him as "support payer". That has go to be one of the most repellent terms I have seen posted here.

