The only thing that I can think of is wait it out. After a bit I would go to the courts and ask for a modification of custody, giving you more and him less. After awhile (more than 6 months) and if nothing has changed, MAYBE you could ask to terminate his rights, but he does have to be informed of all of this. Just because his g/f doesn't like children doesn't mean that you need to take their father away from them... you don't do it, let him mess up not you control everything. Crappy I know, but it'll take time and patience. Just document when children have something going on (ER visits) and write down what he does or doesn't do...