My question involves real estate located in the State of: CA

My neighbor and I installed a new fence between our properties. He suggested that we folllow the old fence line. I agreed, we were in a hurry, and didn't check the boundary lines.. Turned out that after the fence was done, we found a survey pin in the sidewalk showing the boundary (which we had not checked beforehand) and that the fence is on my property. I have a narrow sideyard and want to move the fence (at my expense) to the boundary line (14 inches away on his side). My neighbor does not want to move the fence and claims that I am not allowed, since this fence is the new boundary line and requires both of our approvals.