If you feel that the counselor spoke inappropriately and chose not to mention those things, then doing so now may be seen as a last ditch effort to get your way. If you felt that his comments were not treatment related, you should have mentioned it when your initiated an inquiry about changing facilities. However, you chose this facility and as a PO, I'd refuse your request to change facilities due based on the fact that it will cause an issue with continuity of service.

