closes: #115
related: #65

This PR updates to Kubeflow Pipelines 2.1.0. We have made a small fork of Kubeflow Pipelines, with the only difference from upstream being that ours will work with external S3/MinIO object stores, see https://github.com/deployKF/kubeflow-pipelines for more information about our fork. Hopefully, once kubeflow/pipelines#10625 makes its way into a release, we won't need our fork anymore.

First, please note that this implementation of Kubeflow Pipelines 2.0 is backward-compatible with runs compiled by the KFP V1 SDK (in both v1 and v2-compatible modes).

Second, I would like to highlight that our custom images are still ONLY built for linux/amd64, and are not able to run on ARM yet, see the upstream issue for more information kubeflow/pipelines#10309.

Finally, here are a bunch of technical details about how I got Kubeflow Pipelines 2.0 to work:

• The kubeflow_tools.pipelines.kfpV2.minioFix value has been removed:

  • Setting it will no longer have any effect.
  • With KFP v2, all pipelines now require a ClusterPolicy to inject the object store credentials as environment variables, not just when the embedded MinIO is used.
    • (at least until feat(backend): add namespace & prefix scoped credentials to kfp-launcher config for object store paths kubeflow/pipelines#10625 can be used in a future release)

• The default kubeflow_tools.pipelines.kfpV2.defaultPipelineRoot is now {scheme}://{bucket_name}/v2/artifacts/{profile_name}?region={bucket_region}&endpoint={endpoint}&disableSSL={not_use_ssl}&s3ForcePathStyle=true:

  • this is what allows us to get KFP V2 working with external Object Stores
  • KFP uses the gcloud.dev Go package to access object stores during runs
  • this package has the ability to use s3-compatible servers by setting query parameters like ?endpoint= and reading auth from AWS_ACCESS_KEY_ID and AWS_SECRET_ACCESS_KEY (which we inject with a Kyverno ClusterPolicy).
  • these fixes are the primary reason we have had to make a fork of Kubeflow Pipelines (see https://github.com/deployKF/kubeflow-pipelines)

• you should ignore most changes under ./generator/templates/manifests/kubeflow-tools/pipelines/upstream because it's mostly because we have moved to consuming the manifests directly from kubeflow/pipelines, rather than kubeflow/manifests and they structure their folders differently. Furthermore, we use a lot of patches/removals, so most of those manifests don't actually make it to the final render