My question involves criminal law for the state of: federal tax laws

I found out my ex claimed our kids on his taxes about 6 years ago. He knew I wasn't working and wasn't going to claim them. I have custody, and nothing in our custody papers addresses who claims them (he lives out of state so he didn't see them but a couple of times a year). I called the IRS, and they said he can't claim them unless they are living with him, even if I only had the income of his child support (which he told me gave him the right to claim them). Could he get a charge for this? Should I report it, or has too much time passed?