You got some good advice, and let me add something from actual experience. The situation is not totally analogous, but you can see what has to be done.
My sister's mother-in-law lived in her own home, but with her younger son, who hangs around the wrong crowd. When I say young son, I mean someone in his mid 40's who never worked a steady job in his whole life, and engages in many of the activities you mentioned about your brother.
His excuse for living at home is he helps take care of his own mom. However, after a few falls where he failed to get her to a doctor or hospital, and failure in general to get other things done, like not feeding his own mom, letting her starve, his older brother got involved.
He had to go to court, present evidence, and got himself appointed as his mom's conservator. The stiuation diverges from yours here as he has to sell the family home to pay for his mom nursing home care, and had to get an eviction order as "conservator", so the home can be cleaned up and sold to pay for the care.
In your case, it would appear that the best scenario would be for you to stay in your grandma's home, and take care of her. Depending on guidelines of your state, you can even get Medicaid to send in and pay for a home attendent for several hours a day to help out as we are doing right now for my own mom with Alzheimers.
However, since you are NOT the owner of the home, you cannot force your brother to leave. And since your grandma has Alzheimers, there are competency issue to be first determined by a court, and then appoint someone to take charge, such as you.
You can evict your brother and take charge if you have the legal authority as conservator. Right now, you do not.
And you can gather evidence of what your brother is doing, particularly with the emphasis on "elder abuse".

