Then the issue here is that the passenger said that the drugs were shared by all of you. And you chose to remain silent in regard to that accusation -- certainly your right, but keep in mind that pre-Miranda silence where others might blurt out, "That's not true!" or equivalent can suggest that the accusation is true.
What do you mean by "abandoned"?Quoting StJude
How does that work, if some of the drugs were in the physical possession of the other passenger?Quoting StJude
First part, "I told you I was telling the judge it was mine and none was yours." One issue here is that the passenger also had drugs and that the passenger said that the drugs belonged to all three of you. The driver can say, "The marijuana I held was mine," but even if believed that does not help you with the ostensibly co-owned marijuana held by the passenger. Also, "I'll take the rap for this" is not of itself evidence that the person who is willing to take the rap is telling the truth. Sometimes it's really, "I am going down for this, so even though it was our pot I'll take the blame", and prosecutors understand that to be a possibility.
Second part, "And I told the cops it was mine and you saw they would not let me say it was just mine." Except the police did nothing to compel the driver to change his story. Expressing skepticism does not make you change your story. Further, you told us that it was the rear seat passenger who said "it was all of ours", so the driver's claim that the police didn't believe "It's all mine" would be undermined first by the fact that the rear seat passenger was holding marijuana and second by the fact that the rear seat passenger told a different story. If the driver changed his story when confronted with those facts, it would be reasonable for a judge or jury to conclude that he did so because his original story was not working.
You need to discuss your situation in detail with a criminal defense lawyer and decide if it's worth trying to take the case to trial, or if you should consider other options (some of which may allow you to avoid a criminal conviction).

