Your lawyer can attempt to convince the court that your clothes were clean, and can bring a motion to suppress alleging that the search was improper. Unfortunately I don't expect the "He couldn't have seen marijuana on my clothes" argument to work, because in fact you did have marijuana and a grinder.

Laws are different in each state, and you didn't follow the instruction to identify where this occurred, so consult a local criminal defense lawyer about the availability of dispositions which might allow you to avoid a conviction.