The restraining order is not against you. If you, through your actions, show that you want contact or aren't afraid of the person, that may serve as grounds for modification or termination of the order, or as evidence of why the judge should not punish the person for an alleged violation (depending on the full facts), but as you are not restrained you can legally contact the other person.
Stop talking to this person. If he continues to contact and threaten you after you break off your communications, notify the police.

