My question involves real estate located in the State of: Colorado

We purchased a house that was built in 1947. The neighborhood plat is even older. At one time our house and the house next door were owned by sisters which we believe is the reason the fence zig zags and was arbitrarily placed on the lot. There was also a sidewalk from our backdoor to the neighbor's backdoor. All of the lots in this area are rectangular so the owner of the adjacent property probably knew when he bought his house 5 years ago that the fence was not properly placed.

We paid for a survey and applied for a permit to replace the fence. The neighbor is not so quick to give up what isn't his. Our lot is only 50ft wide and we want to reclaim a 6 ft by 25ft portion.
Does the neighbor have any rights to that strip of our lot? As far as improvements, there is a single shrub that should be on our side of the fence.
Also, we are in the process of building a garage that was placed based on the survey and city zoning acknowledge it was correct.
Any advise would be greatly appreciated!