This PR moves the generation of Kubeflow Pipelines profile resources from Kyverno to explicitly generating them with deployKF templates in the kubeflow-tools/pipelines ArgoCD app.

This is needed because Kyverno (especially the latest version, 1.11.4) has significant performance problems when generating lots of resources (kyverno/kyverno#9633), and there is no real benefit to us depending on Kyverno for generating these resources, as we know the full list of profiles from the user's values.

This also has the added benefit of moving the Kyverno ClusterPolicy (clone-profile-bucket-secret-....) which clones and triggers restarts for the profile object-store secret, into the the pipelines app itself. Because the pipeline ArgoCD app is synced AFTER MinIO, when each profile is first created, there should no longer be a case where the source secret does not exist, except when users are bringing their own object-store. (Note, this is more of a problem in Kyverno versions newer than 1.10.0, which only re-check for missing source secrets each BACKGROUND_SCAN_INTERVAL, which defaults to 1 hour: kyverno/kyverno#9560).

WARNING: when upgrading deployKF:

• this change will cause some downtime in KFP, because the resources will be removed by Kyverno when the ClusterPolicy is removed, only to be re-created when the pipelines ArgoCD app sync happens
• you MUST sync the ArgoCD apps with PRUNE ENABLED, to ensure the old ClusterPolicy is removed and does not conflict with the new explicitly templated versions of the resources
• if you use the automated ArgoCD sync script:
  • you will need to sync twice (respond with yes to the PRUNE prompts). the first sync will always time out when waiting for the pipeline app to become healthy, just sync again after this.
  • alternatively, you can delete the ClusterPolicy before starting a sync:
    • kubectl delete clusterpolicy "kubeflow-pipelines--generate-profile-resources"
• if you sync manually:
  • a single ArgoCD sync with prune enabled should be sufficient for each app