What Factors Will a Court Consider for Custody and Visitation

I am absolutely concerned about my son's well being. It is madness that you would even question that. I am also absolutely terrified of stepping "wrong" in this because I don't want my son to be taken away from me. The father has, in past verbal altercations, threatened to take him from me, as well as having had that threat delivered by others in his family. I am trying to do the best that I can to provide opportunity for a relationship to blossom between father and son and I feel like I am failing, so yes, I am concerned with how I look because of that failure. He has been telling people that "I have been keeping his son away from him" even though he knows where I live, where I work, has my contact information, etc. I am AFRAID of losing my son. I am asking for advice, not to be told my poor child because I obviously don't care because I seem to be more concerned with how I look.
I grew up in a broken home and it was handled god awfully and it left me scarred. I am trying desperately not to have that happen to my poor five month old son because he doesn't know any better and deserves better.

- - - Updated - - -

There is an admitted history of drug and alcohol abuse on his part to our family counselor (when he chose to attend). In past verbal altercations, he has also threatened to take my baby because he was the "more fit parent". He also has a history of domestic violence that he was arrested for. In regards to me there has been a history of verbal, mental, emotional, and sexual abuse, though sadly, the only documentation that I have of it is a journal that I have kept. The father has been marginally present throughout my pregnancy and has had no contact whatsoever with myself or my son from July 9th of last year until late January of this year, even though I know that he has reached out to other people trying to get them to arrange a meeting. As such, I don't feel comfortable with him being alone with my son.

The only thing I have asked him not to do is to show up under the influence. And to smoke outside away from our son. I'm also trying to offer him parenting time outside of the proposed schedule but I am asking for 24 hours notice because I need to schedule a time/place/supervisor for them both.
No my lawyer has not told me that my reasoning for asking supervised visitation is not enough.

You believe the courts may give him custody, even though he never sees the child ? He is trying to frighten you and seems to be suceeding. Don't let him push your buttons. As far as the child abuse goes. It will take more than some old journal entries to prevent him from seeing the child. You would also need evidence, more than your word or that of someone you know regarding the drug abuse. He would more than likely have to do the drugs in the presence of the child. If he isn't seeing the child as scheduled, what makes you think that he is going to see more than originally scheduled ?

He actually admitted the problem to a licensed professional. Does that work towards substantial evidence?
I am not working towards no visitation. That I think is unreasonable. While I am not overly optimistic in this case, I do believe it possible for people to change for the better and be amazing parents. So I think he should have that chance. I personally don't feel like he's starting out on a winning foot but my feelings are immaterial. I can only hope things change because my son really should have a fair shot at his father.

I don't think he will make more of an effort to see his son. I mean, realistically speaking, my son will be 6 months old in 3 weeks and we're batting an average of once in 5 months so far? Those aren't winning odds for my son. At all. But I wanted to at least let him know he has that chance? Does that make sense?