In order for Dad to get custody at this point he'll need to either prove you unfit, or prove that a major change in circumstance of the child's life - like you moving in with a child molester, moving out of state, grades falling dramatically, etc.,
STOP PAYING.
He's bullying you. He can only do that if you allow it - you've got to empower yourself here. You're the one in control - don't allow him that kind of power over you
If he files for visitation, you can request supervised visitation (being that he hasn't seen the child in 5 years) - you will probably get that if requested. If Dad does things right though you must accept that he'll end up with regular visitation if that's what he wants. However I can't honestly see him going through with it.
Sure, he might file something just to scare you again - and really you know him, we don't - but do you think he's going to turn up to all of the court dates and go through with it all?

