You are obligated to give him whatever rights and access he has under your rental agreement, written or otherwise. (Read this.) If you want to work things out with him, then work things out. Otherwise terminate his residency and find a different lodger.

