You were terminated so why wouldn’t that be a proper response?

I don’t really see your actions as a refusal either. It was, in my opinion, a reasonable inquiry based upon your previous weed pulling activities. You could have phrased it differently but I don’t see it as any wrong doing.