Look, you gave him the keys and told him he could drive it. He had the car with your permission. Maybe if you had expressly told him he could not drive it for any other purpose than driving locally you might successfully argue that he was driving without permission. But you didn’t do that, apparently. You are simply trying to say the limit on use of the vehicle was implied based on what he told you he planned to do, and that’s not likely to work. In any event, you need to tell the insurance what the situation is and it would make the call as to whether it will pay any claim. This is not really the big deal that I think you fear it is.