He can demand whatever he wants. He can contend that you are only connected to the fence over his property with his permission, and that he has the right to revoke that permission. You can counter that there was a binding agreement, evidenced by your payment for half of the (almost) boundary line fence, that he cannot unilaterally revoke. If he takes it to court, a judge will evaluate the claims and evidence and make a decision.
It's unlikely that he will take it to court given that he's selling - this is probably primarily about removing anything which might interfere with the sale.

