This is a private process server? Self-employed or an employee of a company? A court officer? If he works for somebody, he has a boss or supervisor to whom you can complaint.
The time to object to improper service is prior to the resolution of your case. I don't know whether the landlord's lawyer was telling you not to raise the issue because you and he decided to resolve the matter with a stipulated order and, as is typical for housing court matters, he had other cases to worry about, or if he knows the process server is dodgy. But one way or the other, you missed your best opportunity to raise the issue in court.
I expect that, if called to testify, the process server would state that they described the adult at your home who answered the door and that they never claimed that the person was you - just an adult fitting the specific description who refused to provide a name. At most you're going to get a, "Perhaps I made a mistake and served apartment 15 instead of 19 - the five is smudged on this document and looks a bit like a 9."

