I think you should consult with an attorney to see if the attachment is an official part of the decree. If your ex-husband tries to pull your child back to him, you could go to court for an official child custody order. Given that he has not been in the child's life often, I think any legal dispute would favorable for you. Nevertheless, I would recommend you share your story with a lawyer to confirm this.