How long ago was that fence installed? He may have a valid claim for adverse possession of the property.