Can I File Child Abandonment

My question involves child abuse or neglect in the State of: Florida
My son was born in the state of Ms, his father left when I was 35 weeks pregnant. A lawyer in the state of Ms stated this was considered abandonment and I could move out of state w/o the fathers permission. I moved to Florida. I have never pursued child support or any other support from his father. In Jan 2012 he sent me a request stating he wanted to see his child. I allowed it. It didn't end well, he was drunk and admitted to buying street drugs. He hasn't sent any child support money nor have I asked for any, he occasionally sends me a message on FB asking how my son is and asking if I could call and let my son talk to him. He states I am keeping my son from him and he has rights to him. I don't feel I am, but am I somehow breaking the law? I am doing my best trying to raise a child on my own w/o any help.

What do you believe it means to "file child abandonment"? Leaving the child with the other parent is not "child abandonment" in the sense relevant to abuse and neglect / protective proceedings.

From what you have told us, dad has no visitation rights and pays no support. If he sues for visitation, you can present your evidence in opposition to his request, and can ask that he be ordered to pay support.