My question involves a child custody case from the State of:Florida

I need some serious help. My daughter is 3.5 years old. She has not seen her father in 3 years. We were married. He did not want anything to do with the pregnancy and this caused the divorce. Through a mess up at the hospital he ended up on the birth certificate. We went to court and he was awarded the generic every other weekend visitation. Which he NEVER used. He has seen her about a 7 times total till she was about 6 months and then quit.
A year later he asked the courts to allow him to see her again. They pulled his medical records and found him to be homicidal/suicidal and bipolar. The judge granted him 2 hours of supervised visitation every other weekend. This was TWO years ago. He never saw her. He had his lawyer send a letter stating he wasn't interested in seeing her.
He was asked to pay $1,000 child support a month. He hasn't paid any of this. (which I don’t care, I am fine without his money) When we went back to court, the judge lowered it and hit him with arrears, which he never paid. I tried to get him served with contempt of court but he dodged the process server for months till I said forget it. I have blown $15k in attorneys fees to protect her and am now broke with no lawyer.
My daughter has severe ADHD and I can't make any major medical decisions without him. I can't get her a passport to travel without him, I can't move. All I want to do is protect my daughter and move on with my life.
What constitutes as abandonment in Florida? What are my chances of asking a judge to remove his rights and them doing so? What if he comes to court and just to be an ass (because he is) he then asks to have contact with her? My boyfriend and I have lived together for a year and this is all she knows and loves. Im petrified of my ex and his medical records and such. I really need help!!!