It doesn't matter whether you were using the benefits or not. UNDER THE LAW they may not allow you to drop them unless it is either a qualified life event or the annual open enrollment period, and the IRS, not the employer, decides what is a qualified life event. The employer would be in violation of the law if they allowed you to drop the benefits when it was not a qualified life event (or open enrollment) and if caught in an audit (and ALL plans are audited regularly) it could put the entire plan in jeopardy. I am not exaggerating.
Sorry, but you do owe the money.

