You allowed it. It wasn't a problem. Is there anything in the court order that said that the child MUST live with either parent? If so, YOU "broke" the court order as well.
You're not going to keep the child from mom. If grandmom pushes the custody issue, mom only need to go get her child.

