While he might provide sole financial support, he is not the primary caregiver - he works, and you've stayed home looking after your son. This makes you the primary caregiver in the eyes of the court, and generally speaking the court prefers to leave things as they are unless there is a question of fitness.
One more question - what is your current status, immigration-wise? Given that you were already married two years before you came here, you should have received your unconditional green card, yes?

