Independent consultant wants to continue to work under agency contract or at least wants client who is forcing the change to get the agency to release the contractor - this change was never our choice
as I said; the contract you need to be concerned with is your contract with the agency. If that doesn't restrict you from working directly for a prior client, then you are good to go.

As to the contract between the agency and the client; that is out of your hands. If that contract restricts the client from hiring a person supplied by the agency, then one would presume they will attempt to enforce that contract.