Does your court order say anything about a known 3rd party picking child up at meeting place? Or can just father do the pick up?

If the answer is that a 3rd party can pick child up, then I say you need to file for a modification asking for either a ROFR or that his days be cut due to his job.

ETA: His days be cut, but offer to make them up during a vacation period he may have coming up or Christmas time, or something of that sort. You should still want the child and father to have a relationship even if Dad has a crappy work schedule that makes him miss his weekends.