No, that's not correct.
Aardvarc was simply pointing out the fact that he hasn't been established as father. In your State, this must happen before his rights - which haven't been established yet - can be terminated.
Also, he has had contact. Either way, you cannot claim abandonment.
With regards to the drug use, you might find yourself in a pickle. If you have photographic evidence that you haven't submitted to the authorities - why haven't you done so? You've also allowed him to have contact even though you know that he does drugs. The court might wonder why you were "ok" with it up until now.
Honestly - in your situation you can't afford to do this without an attorney. I can absolutely see Dad fighting this.