Entirely up to you.

The whole point of at-will employment is that you can quit for any reason at all, and you can be fired for any reason not expressly prohibited by law. This INCLUDES no reason.

No laws have been violated. It's your choice whether you want to accept his firing you, have the meeting with your boss, or what.