My question, given I allowed fence to be placed on my property does this somehow change ownership/property line rights?
Sometimes a fence is just a fence. Yours is what I would call a "convenience" fence.

As it was built according to your permissive use, it is very unlikely that the location can ripen into a claim of your property.

Permission is just that and can be withdrawn by you at your "convenience".

The neighbor may have a change of heart after finding out what the survey will cost. It may just die down. But I think that you need to deal with this in a proactive way as, in the long term, time will not be on your side.

If the neighbor does not follow through with a survey, I would suggest that you have your attorney send them a letter stating your take on the fence, how it was placed with your permission, and that you are continuing the permission until further notice. Your attorney will know what to say.

If the whole thing descends into a "he said she said" squabble over the location of the boundary, somebody will be paying for a survey.