It is my understanding that unless your PD is felon, a visible drunkard (i.e., shows up drunk for court) or is openly lackadaisical about your defense, no, you can't fire him.

You have to have very solid reasons to ask the judge for a new PD, and absent something heinous, your chances of convincing the judge to assign a different PD are very slim to non-existent.