If your anxiety is not relevant to your job, then you made a mistake in discussing it with anyone at your job. That was the only way they knew of it, after all. And when you cited it as the reason they should do something for you, you opened yourself to what they have requested. There is no other way to see this any differently than that.