Basically you want to remove him from the board if he is so unpopular. See Section 718.112(2)(j)
http://www.leg.state.fl.us/Statutes/.../0718.112.html
However I really recommend to you that you think twice about this. He will be in your face about everything. He will threaten and intimidate you all the time. You will not be certain and you will worry if you did something slightly not according to law that he can sue you for. And there will be legal fees because you will often be tempted to contact an attorney to make sure he can't hurt you. If he's been board member for awhile and the sky is not falling and the association is not on the verge of bankruptcy and your assessment is not double that of similar associations in your area, you may be better off letting the bully rule.
Either way, it may be best that you contact the association attorney to guide you through the removal. You also better be certain that half the unit owner will vote for removing him. Otherwise you'll just rack up legal bills and won't remove him.
As for him hinting a lawsuit, anytime someone tells me this, I say to them well if you feel that this is best for you go right ahead. Still no lawsuits :-). People bluff this one all the time. But it takes a strong gut to stand up to this....