I'm confused about it taking 4.5 years for it to go to collections. Do you know if the card had a balance at the time of the divorce? If so, it would have been in collections years ago unless someone was paying the bill for 4 years and just suddenly stopped.
If it had no balance, then someone obviously used the card and didn't pay it.
Either way, I don't see how he could hold you responsible for his personal debt if it wasn't listed in the marital debt. I agree with you that if the collection agency sent the letter to him, then the original bill would have been connected to his address also.

