You have no say as to whether charges are pursued or not.

No, you cannot invoke spousal privilege. Yes, you can be compelled to testify. If you avoid the subpoena or refuse to testify you can go to jail.

Also, if you refuse to cooperate, if you and your husband have any children you can expect child services to get involved anbd potentially seize the children or impose conditions for the two of you to attend counseling or anger management.

Ignoring this is not going to make it all better. Typically, these situations get worse, not better.