You would not want to unilaterally stop the visitation--the proper way to address the issue would be to seek a modification of the existing court order.
If the noncustodial parent, while exercising visitation, has others around the child who are not acting in the best intereets of the child, you could seek a modification for supervised visitation or for a prohibition that certain individuals could not be around the child during the exercise of vistitation, the violation of which could result in contempt and/or termination of visitation rights.
Use of the drugs by the noncustodial parent is definite grounds for supervised vistitation.

