The officer should not be on your property without reason. However, you not being aware of his reason, doesn't mean that he didn't have one. With the officer leaving abruptly I think that either he didn't have a valid reason, or he did and didn't want to tell you.
You can contact the PD and ask them. Depending upon the ethos prevailing there as to whether you get an acceptable response.

