If you don't pay him and ask him to use the security deposit ( assuming it's as much as the rent), then he really can't do too much.
It's not like he can say FU, throw your stuff outside, and change the locks.
Can he file papers, and take you to small claims court?
Sure, but is it really worth the effort?
No.
But keep in mind, if there are major repairs to be done, and he needs to use the security deposit to do the work, then you will be in court prety quick.
And he will prolly get you for court costs as well