How old is your brother? If he is under 18 then you will be forcing him to go stay with mom when dad goes away.
A court is likely to look at this and remind you that you are an adult and can legally leave when mom comes over to watch your brother. If you do manage to obtain a restraining order of some kind, your brother will either have to be left in YOUR care, or he will have to live with her for a week to a month when your dad travels.

