My question involves real estate located in the State of: CA

I built a small retaining wall as the base of a new fence on a hill behind my house on what I thought was the property line. It turns out that the new fence crosses onto my neighbor's property for about 50' starting at zero ending at around 18" over the property line. At first he said the fence would be fine because it was unusable land to him. I suggested that we write up an irrevocable easement and offered to give him the equal amount of property in easement in an unusable portion of my land. Even though he said the fence was fine, at the end of the day He didn't want to put it in writing.

Next he sent a very threatening, legal, certified letter asking us to remove the portion of the fence that was on his land. I told him I'd like to get my own survey to verify exactly where the property line was before going to the expense of taking down the wall. He sent me a nasty email saying that he was going to have his contractor put a chain link fence where he thought the property line was and that he would take care of fence I built. I asked for his patience for me to verify where the line was before we moved anything.

Although it turns out his survey was never recorded, my surveyor told me that his survey is close to correct. Now that I'm clear where the fence crosses over onto his land, I told him I was ready to remove the fence at my expense. He seemed concerned about liability issues associated with the demo and removal, so I wrote up a release of liability and permission to access his property for the purposes of removing the fence. I also wanted him to agree that once I removed the wall, he couldn't come back and hold me liable for anything related to the construction or demolition of the fence.

Today he said he didn't want to sign the document and didn't want the wall removed. He wants me to build my new fence on my side of the property and he wants to use the wall I built. I have hired a bonded, licensed contractor to take the fence down in one day, but it would require demolition of concrete that could fall further onto his property. What could happen to me or my contractor if we went ahead and demolished the block wall and fence that falls on his side of the property line?

I don't trust him since he has backed out on everything he said he would agree to so far when I put it in writing. All I want is resolution and to absolve myself of liability with this guy. I don't want him to own the fence I built. Please help.