Your best option may be going to court and make your case that Dad's rights should be terminated as part of a stepparent adoption.

Mom, I'm terribly sorry for your dilemma. And honestly, you need to speak to a local attorney - desperately. AZ doesn't grant stepparents many rights at all (though stepparent visitation is possible), for the most part, and while AZ does recognize the concept of the de facto parent, current case law indicates that this is applied more commonly to LGBT couples rather than hetero couples.

(The reason being, that the hetero stepparent has another remedy available - that of visitation).

Please, speak with a local attorney. I feel a stepparent adoption (four years of not visiting his child might convince a court) might be your best, and easiest, option. That would mean that your husband is your child's legal parent. Your ex - Dad - would have no rights or obligations at all.