..and you're not going to be granted sole custody based on the fact that you want to get the child a passport.

Just call Dad, ask him to sign the form and have it notarized, send it back to you and go apply for the passport. Not a big deal.