Very simply, "the court will rarely act on "ifs and ands" "
In other words - the legal parents have the RIGHT to parent their child. Unless there is actual proof of imminent harm, the court doesn't want to listen to "oh s/he is such a bad person and what if.....?"