My question involves unemployment benefits for the state of: Maine (I will admit I am not a fan of this state)

I was laid off from my job about a year ago, filed for my unemployment benefits and after getting "stuck" in the system a couple of times I got my benefits with no problem.
I took a part time job for 3 weeks, made under $500, the wages were not reported by me.
Had a hearing with a deputy, I did not deny I had the job, I did not deny wages, I explained it appeared I misunderstood a question he kept pointing out which read "Did you earn wages including self employment and commision" now I am pretty literal, and the State of Maine seems to ask the same question in different flavors, for example, they will ask "Did you receive any severance pay, vacation pay" then they will ask if you received terminal pay, or pension benefits. It sounds crazy but remind you I am only a 5th grade level of reading so I failed to report part time wages. So I told the deputy that was my reason, he closes the call the same way he opens it by stating that he will review the information (this was not a recorded call) with all the questions I had to answer the state was not interested in my part time earnings (I only worked the job for a month) just like the question does not ask if about money I earned if I sold my goats crap on ebay for $50. If the state wanted to make things easier for everyone they should consider these mistakes. When he closed the call he immediately said I was guilty of fraud to the system and I will have to pay back some benefits, I immediately told him he just said he was going to close the call and review things, his words were that he did not care, and he can do whatever he pleases...yes he said that.

So he mailed the decision that states I owe the state thousands of dollars after penalties and am not eligible for another year, I appealed to the State administrative office, I explained my position on the understanding again and also said it was wrong to be called a fraud on the first hearing, I have a clean criminal record and when you look up fraud, hear about it, that is pretty big in my eyes. They asked if I received a booklet about how to file my benefits, I told them I did not, they asked why I did not call and tell them I didn't I told them, there was no checklist enclosed with my benefits so how was I to know I was supposed to get this blue book he spoke of. He contested that back in 2004 I reported earnings, I argued back those were full time wages, the hearing officer set that aside then the deupty stated I reported wages at the beginning of the year, I told him I did but that was severance pay by the employer and my last week of pay they were witholding until I signed a piece of paper agreeing to never defaming the company for laying me off. I then also stated that according to the call center that I filled out this information on the wrong field and had to fix it, which was no big deal to them, so even though he said I did it right, I told him I was told I did it wrong and what held me form getting my benefits to begin with.
The hearing officer closes the case, sends an answer that continues my disqualification but shortened it to April 2012 and demanded a repayment.

I then filed an appeal to the State Insurance Commissioner, I waited 3 months after submitting a statement that reaffirmed my misunderstanding and nobody has yet to prove that I am frauding the system intentionally, I stated if I received a payment during my part time brief period, I had no problem paying just that back, but I disagreed with the disqualification, after 3 months they sent me a letter that just states they are adhering to the decision, they did not state what lead them to that conclusion, they did not have a hearing, they did not state if they listened to recorded calls, written statements, etc just they agreed.

Now they give me 2 choices it seems. Ask for reconsideration or go to superior court in my county and file a hearing request with the state judicial system. I am a bit aggravated over something misunderstood and a question that is constantly repeated. The state is OK to make a mistake by having me stuck in the system, not sending work search logs but when I have a misunderstanding it is a whole different game. Personally I feel that the system is not fair, everything is "State of Maine" and they are all in it to not release money to anyone the second they slip.

I have tried to explain I have no problem repaying the under $500 I failed to report but also and probably wrongfully stated the questions could be summed up as one question. The deputy was asked on my second hearing why the department of labor seems to ask the same question repeatedly, he stated that is because the first $25 in some cases depending on the type of earnings received can be excluded and not deducted from your upcoming unemployment benefit, however I did find in the states website that they do say in a handbook it does not say that, it just says if you earn more then $25 (does not say how) they will automatically overlook it, it does NOT state it depends on how you got it as the Deputy tried to argue.

I am fed up, I want my benefits back, I am not sure at this point if I should go to court or try to convince a reconsideration and inform the state to resume my benefits and withold what I owe. I made it clear it was an honest misunderstanding and I have challenged their claim to how I knowingly committed fraud (that is what they keep saying in reference to the law) What should I do? Is it worth taking this to court? Is it OK for the state to make mistakes but crap on the little guy when he does? Is it worth trying the 3 member commissioner a reconsideration one last time and emphasize paying back what I owed? I thought I been pretty nice and polite to them even, I guess I need to bake these guys muffins or something to show I am nice....