My question involves landlord-tenant law in the State of: NC

Landlord is asking now all of a sudden when I am moving out that I pay all late fees or it will be deducted from security deposit. I never saw, but the lease talks about it, but the landlord has NEVER once reminded me of it OR charged me late fees so I have been paying rent whenever I could. I have had my lease even extended previously, so why didn't landlord tell me this at that time if it was due? Is this even lawful?, can I refuse to pay it and then sue landlord for deposit not being returned to me?