I think you could wind up involved as witnesses, but the action itself is usually limited to being against the paramour, the person stealing away the spouse.