At the time you married he was still married and so the marriage cannot be considered legal.

He broke the law by marrying you while still married. It shouldn't be difficult to have it annulled especially since you haven't been together.