In the relationship between a patient and psychiatrist providing private and confidential information is therapeutic. Setting up a trusting therapeutic relationship usually takes time. The distrust you describe is likely to be due to giving too much information to someone you don't trust.
It is impossible to predict the real damage that could come from this. But I know it is not unusual for a patient to be concerned of harm when they distrust their doctor. So I encourage you to continue to find someone you trust.
You know the reason the movie "Aliens" was so popular? You are not the only person who has imagined the things we fear are "in us and all around us".