I believe the law states that an officer must have reasonable suspicion to conduct a traffic stop.
He stated that he didn't know if you were coming off the wrong way or had just made a U-turn. He had reasonable suspicion (backed up by your testimony). He accepted your answer and let you go.