You didn't trim, you eliminated stumps generally don't grow back into trees, they stay stumps. Additionally, there is a vague permission in the conversation between the husbands and not explicit permission for what was intended. I did note that in your original post you state that there were several trees but when it came to replacement you say there was only one. Which was it?

You are responsible for putting it back the way it was and no more.

A survey would probably be a good idea but it won't easily make this issue go away. Just because you obtain a survey that (may) indicate that these trees were in fact yours doesn't mean that the neighbors accept it and continue to push.