Custody change

I have been divorced for 5 years and i have full leagl physical custody of my children. I have recently remarried and also have one child with my husband now. My ex still lives with his parents and has now moved in his girlfriend. He has visitaion everyother weekend. Since his girlfriend has moved in I decided not to let the children spend the night for a few reasons. First they all sleep in the same room. second shes a known drug user. Third she gave up rights to her own child.
My ex has forgotten the kids twice in one month and now he is telling me that he is going to go to court and get joint custody of the kids so i cant tell him what to do.
My husband and I make Triple what he does, the kids have lived in the same house for 5 years. They have went to the same school for 4 years. Atteneded the same church since they were born. Is my ex able to change the custody and do I have the right to stipulate when he sees the children now?

You don't have the right to unilaterally refuse parenting time, in defiance of the court's order. If you don't like what the court ordered, you should petition the court to modify it. And yes, your refusal to obey the court's parenting time order can provide the other parent with ammunition in an effort to modify custody.

His Visitaion was left up to me. Just had to be reasonable. Him and I decdied on the everyother weekend thing. Like I said He's forgot to pick them up twice. He still gets them just not over nights anymore.

It sounds like you have issues with your ex husband's new relationship. It would be better for all the grown-ups to act like grown-ups.