You are controlled by your contract. If your contract limits your business with the client, you are bound by it regardless what the issue is between the agency and the client. Even if there was just cause to terminate the contract between the client and the agency, it does not affect your contract with the agency.

