Is there hostility between the parents of the child?
Is there hostility between the mother and you?
The reality is that you are NOT a grandparent, de facto or otherwise, and there is no "legal" basis for you to claim so.
If you and your former "ward" are not on the good side of the mother then you will just have to resign yourself to occasional visits when the father has visitation. Perhaps you can encourage him to increase his visitation but you cannot compel it nor do I think any judge will rock the boat because of your feelings.