Laws vary by state. In a defamation case it's normally up to you to prove that the statements are false. If he's accusing you of criminal acts, in many states that would constitute defamation per se and reverse the burden of proof, such that he would have to prove that his statements were true.
If the statements are true, you're probably going to make things worse for yourself by suing.

