Do you have any court orders defining custody?

You cannot get him removed from the birth certificate unless paternity is legally disestablished in court. Depending upon the nature of the crime, you MAY be able to get his parental rights terminated in court and you'd need an attorney for this.

Other than that, the State Department requires that either you get Dad's consent or that you have sole legal custody in order to get a passport.