feat(backend): Upgrade argo to v3.4.16 by gmfrasca · Pull Request #10568 · kubeflow/pipelines

gmfrasca · 2024-03-13T23:43:06Z

Description of your changes:
Implements #10469, also Fixes #8935

Upgrades Argo backend to v3.4.16, latest in the argo v3.4 release stream
Implements RetrievePodName function to work around NodeStatus.ID issue (see below)
Updates unit tests accordingly

Note: This PR supercedes #9301, copying description from PR:
Fixes the following CVEs:

https://github.com/advisories/GHSA-4f9f-mpmj-4c52
https://github.com/advisories/GHSA-98w6-hw73-ph8m
CVE-2022-23521
CVE-2022-41903
https://github.com/advisories/GHSA-grfr-78m7-q35q
https://github.com/advisories/GHSA-cfmr-vrgj-vqwv
https://github.com/advisories/GHSA-75qm-2q4j-qx6g

Breaking changes:

- Argo 3.4 removed support for choosing container runtime executors, emissary is the only option left: https://argoproj.github.io/argo-workflows/workflow-executors/
- Argo 3.4 no longer maintains the naming convention of NodeStatus.ID and NodeStatus.Name being identical.  This was addressed in the code changes in `workflow.go`

Notes:

The fix for the NodeStatus naming bug/breaking change was extracted to a seperate function, such that when it is fixed it can be easily updated in one place. Monitor the fix in argo here: feat: store podname in nodestatus argoproj/argo-workflows#12503
I believe the containerRuntimeExecutor value in the WorkflowController ConfigMap needs to be removed in all manifests, or else the WC Pod will fail on an invalid input json error. May need assistance testing across all platforms

Checklist:

The title for your pull request (PR) should follow our title convention. Learn more about the pull request title convention used in this repository.

google-oss-prow · 2024-03-13T23:43:18Z

Hi @gmfrasca. Thanks for your PR.

I'm waiting for a kubeflow member to verify that this patch is reasonable to test. If it is, they should reply with /ok-to-test on its own line. Until that is done, I will not automatically test new commits in this PR, but the usual testing commands by org members will still work. Regular contributors should join the org to skip this step.

Once the patch is verified, the new status will be reflected by the ok-to-test label.

I understand the commands that are listed here.

Details

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

HumairAK · 2024-03-14T19:29:36Z

/ok-to-test

rimolive · 2024-03-14T21:44:26Z

/test kubeflow-pipeline-e2e-test

gmfrasca · 2024-03-14T22:42:31Z

I believe the CI errors are due to the respective Argo images not existing in the gcr.io/ml-pipeline image repository (WorkflowController pod fails to come up because of ImageBackOff error, since the image doesn't exist), since this PR would be the one that updates them.

I'm assuming I don't have permissions to build and publish them to the repository so who would be responsible for taking that action?

gmfrasca · 2024-03-14T22:48:56Z

Another quick note: When testing this, there was a unit test that was failing (Test_executeV2_Parameters), but this was also the case before any changes made by this PR, so presumably that's out-of-scope here. All other unit tests should pass

rimolive · 2024-03-15T13:37:58Z

@chensun @zijianjoy Can you push Argo 3.4 images to gcr to ensure test will pass?

chensun · 2024-03-26T18:43:22Z

@chensun @zijianjoy Can you push Argo 3.4 images to gcr to ensure test will pass?

Done. gcr.io/ml-pipeline/argoexec:v3.4.16-license-compliance and gcr.io/ml-pipeline/workflow-controller:v3.4.16-license-compliance are now available.

#10618

chensun · 2024-03-26T18:43:32Z

/retest

rimolive · 2024-03-27T11:21:05Z

/test kubeflow-pipeline-backend-test

gmfrasca · 2024-04-03T19:49:24Z

looking into the e2e failure but in the meantime, looks like mkp was a network flake so retesting to see if investigation is needed

/test kubeflow-pipeline-mkp-test

Signed-off-by: Giulio Frasca <[email protected]>

- Argo 3.4.16 upgrade introduces a breaking change with inconsistent node.ID vs node.Name - introduce a function in workflow.go to conditionally handle this Signed-off-by: Giulio Frasca <[email protected]>

- PNS Executor was removed in Argo v3.4, so manifests no longer valid - WorkflowController will fail to start if `containerRuntimeExecutor` provided as input parameter, so remove from WC ConfigMap and CM patches Signed-off-by: Giulio Frasca <[email protected]>

- Stemming from upgrade to argo 3.4, Pod Name is no longer always the same as NodeID, which breaks a few tabs (PodInfo, PodEvents and PodLogs). Add function to address this Signed-off-by: Giulio Frasca <[email protected]>

Signed-off-by: Giulio Frasca <[email protected]>

gmfrasca · 2024-04-16T14:35:30Z

^just addresses the rebase/merge conflicts (only affects go.mod and licenses CSVs)

HumairAK · 2024-04-16T16:39:33Z

@chensun / @zijianjoy bump, any thing else we can do to help get this PR merged?

rimolive · 2024-04-16T16:40:39Z

/lgtm

chensun

/lgtm
/approve

Thanks!

google-oss-prow · 2024-04-16T16:51:25Z

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: chensun, Tomcli

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Details

Needs approval from an approver in each of these files:

~~OWNERS~~ [chensun]

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

google-oss-prow Bot added the do-not-merge/work-in-progress label Mar 13, 2024

google-oss-prow Bot requested review from Ark-kun, chensun, dushyanthsc, gkcalat, jlyaoyuli and zijianjoy March 13, 2024 23:43

google-oss-prow Bot added size/XXL needs-ok-to-test labels Mar 13, 2024

gmfrasca force-pushed the argo-3416 branch 2 times, most recently from 93933f8 to dd939f3 Compare March 14, 2024 00:05

chensun requested a review from Tomcli March 14, 2024 01:00

google-oss-prow Bot added ok-to-test and removed needs-ok-to-test labels Mar 14, 2024

gmfrasca mentioned this pull request Mar 14, 2024

WIP: UPSTREAM: <carry>: feat(backend): Upgrade argo to v3.4.16 opendatahub-io/data-science-pipelines#28

Closed

1 task

gmfrasca changed the title ~~WIP: feat(backend): Upgrade argo to v3.4.16~~ feat(backend): Upgrade argo to v3.4.16 Mar 26, 2024

google-oss-prow Bot removed the do-not-merge/work-in-progress label Mar 26, 2024

gmfrasca force-pushed the argo-3416 branch 2 times, most recently from fe7c39f to b2fbc1c Compare March 26, 2024 22:05

gmfrasca force-pushed the argo-3416 branch from b2fbc1c to 3774927 Compare April 3, 2024 17:21

google-oss-prow Bot assigned Tomcli Apr 12, 2024

This was referenced Apr 14, 2024

[frontend] ui can't retrieve pod logs when using argo-workflows 3.4+ (POD_NAMES change) #8935

Closed

feat: store podname in nodestatus argoproj/argo-workflows#12503

Closed

gmfrasca added 7 commits April 16, 2024 10:26

pull argo v3.4.16 upstream

c5d22d5

Signed-off-by: Giulio Frasca <[email protected]>

upgrade to Argo v3.4.16

490dfd4

Signed-off-by: Giulio Frasca <[email protected]>

Update ValidateWorkflow calls

46256a2

Signed-off-by: Giulio Frasca <[email protected]>

Add NodeStatus pod name retrieval function

a95e1cb

- Argo 3.4.16 upgrade introduces a breaking change with inconsistent node.ID vs node.Name - introduce a function in workflow.go to conditionally handle this Signed-off-by: Giulio Frasca <[email protected]>

test: update frontend CI to accommodate pod id/name changes

3e33f2f

Signed-off-by: Giulio Frasca <[email protected]>

gmfrasca force-pushed the argo-3416 branch from 80179cb to 3e33f2f Compare April 16, 2024 14:33

google-oss-prow Bot removed the lgtm label Apr 16, 2024

google-oss-prow Bot added the lgtm label Apr 16, 2024

chensun approved these changes Apr 16, 2024

View reviewed changes

google-oss-prow Bot assigned chensun Apr 16, 2024

google-oss-prow Bot added the approved label Apr 16, 2024

google-oss-prow Bot merged commit 809d576 into kubeflow:master Apr 16, 2024

rimolive mentioned this pull request Apr 16, 2024

[backend] caching fails when task pods have WorkflowTaskResults RBAC (argo 3.4 change) #8942

Closed

This was referenced Apr 18, 2024

Add gmfrasca as member to kubeflow org kubeflow/internal-acls#672

Merged

Add gmfrasca as member of Kubeflow org kubeflow/internal-acls#673

Closed

thesuperzapper mentioned this pull request May 3, 2024

feat: update to Kubeflow Pipelines 2.1.0 deployKF/deployKF#122

Merged

HumairAK mentioned this pull request May 11, 2024

[feature] Upgrade to latest Argo 3.4.x #10469

Closed

AndersBennedsgaard mentioned this pull request May 27, 2024

[frontend] logs and artifacts stored in different Minio locations and can't be accessed by frontent #6428

Closed

terrytangyuan mentioned this pull request Aug 6, 2024

Add support for ppc64le argoproj/argo-workflows#12449

Open

agilgur5 mentioned this pull request Sep 10, 2024

Kubeflow Pipelines: Non-transient error: failed to resolve {{ppln-from-vsc-xkhhr}} argoproj/argo-workflows#13512

Closed

4 tasks

kimwnasptd mentioned this pull request May 27, 2025

Exploration: Investigate how POD_NAMES of Argo can affect KFP V1 canonical/bundle-kubeflow#1268

Closed

Uh oh!

Conversation

gmfrasca commented Mar 13, 2024 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

google-oss-prow Bot commented Mar 13, 2024

Uh oh!

HumairAK commented Mar 14, 2024

Uh oh!

rimolive commented Mar 14, 2024

Uh oh!

gmfrasca commented Mar 14, 2024 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

gmfrasca commented Mar 14, 2024

Uh oh!

rimolive commented Mar 15, 2024

Uh oh!

chensun commented Mar 26, 2024 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

chensun commented Mar 26, 2024

Uh oh!

rimolive commented Mar 27, 2024

Uh oh!

gmfrasca commented Apr 3, 2024

Uh oh!

gmfrasca commented Apr 16, 2024

Uh oh!

HumairAK commented Apr 16, 2024

Uh oh!

rimolive commented Apr 16, 2024

Uh oh!

chensun left a comment

Choose a reason for hiding this comment

Uh oh!

google-oss-prow Bot commented Apr 16, 2024

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

5 participants

gmfrasca commented Mar 13, 2024 •

edited

Loading

gmfrasca commented Mar 14, 2024 •

edited

Loading

chensun commented Mar 26, 2024 •

edited

Loading