Apache Airflow Provider(s)

amazon

Versions of Apache Airflow Providers

apache-airflow-providers-amazon | 2.4.0

Apache Airflow version

2.2.3 (latest released)

Operating System

Amazon Linux 2

Deployment

MWAA

Deployment details

No response

What happened

Sagemaker Processing Operator no longer honors the action_if_job_exists param and always fails creation of a new processing job is a job with the name already exists.

This happens because in a recent change, the function responsible for executing the job no longer honors the increment setting:

Change that breaks the increment: 96dd703

New code:

What you expected to happen

When Sagemaker Processing operator is called with a job-name that already exists, the job creation should succeed with a name that is incremented by 1.

How to reproduce

invoke SageMakerProcessingOperator twice with the same job name while keeping action_if_job_exists as 'increment'.

Anything else

No response

Are you willing to submit PR?

• Yes I am willing to submit a PR!

Code of Conduct

• I agree to follow this project's Code of Conduct