Yes you can.

But Dad can walk into court and demand the child is returned pending a hearing addressing relocation.

Do it the right way; California is currently "pro" moveaway - meaning that the non-relocating parent must show why the move isn't in the child's best interest. Given the age of your child and Dad's lack of interest I don't see him being successful. So file now and get it all approved by a judge.