Accused of Debit Card Fraud

My question involves criminal law for the state of: New Hampshire

My son is accusing me of fraud of 10,000 over the past 4 years. My son had given me his PIN number to help me out and had told me if I ever needed it I could use it. Yes I have used it without asking him but since he had told him I would be able to use it if I needed it, I didn't bother asking his permission. I never kept his card with me, always used it when he was home. Sometimes he would come over and asked me if I had used his card and I would tell him yes I needed it to pay groceries or rent and he would leave it and would not discuss it any further, but now he is saying that he never said such a thing. To repay him I would asked if he needed anything or I would pay for some of his car repairs or bring him here and there. Now that I was going to start a new life he decided to accuse me of Fraud. I am scared that I will be sentence and have jail time... Is there anything I can do to avoid all of this ?

You can pay him the money that you owe?

How could your son possibly not have known that you spent $10,000 out of his bank account over a period of four years?