
Quoting
pg1067
What is your work relationship with this person? Are you his supervisor or in any way a senior employee? If you are in any way senior to him, do you have a history or making in appropriate comments of this sort?
Subject to how you answer those questions, the answer to yours is that the absolute worst thing that would happen is you might get fired. You might also incur several thousand dollars in attorneys' fees to get the lawsuit dismissed (unless he sued in small claims court, in which case you'd be representing yourself). Any lawsuit he might file would be frivolous.
That said, and while I obviously don't know what sort of work you do, the question you asked is beyond inappropriate for any workplace.