Skip to content

Comments

Support AF3 connections, variables and pools CLU#1869

Merged
jaketf merged 1 commit intomainfrom
feature/support-af3-connections-variables-pools
Jun 16, 2025
Merged

Support AF3 connections, variables and pools CLU#1869
jaketf merged 1 commit intomainfrom
feature/support-af3-connections-variables-pools

Conversation

@feluelle
Copy link
Member

@feluelle feluelle commented Jun 13, 2025

Description

Describe the purpose of this pull request.

Support AF3 connections, variables and pools CLU

  • update airflow client to be api version agnostic
  • use metadata airflow api url for airflow client
  • add include_deferred to airflow client pool schema
  • add include-deferred cli flag to pool create
  • add include-deferred cli flag to pool update as it is required for AF3
  • show include deferred in pool listing

Note

This PR removes support for pools CLU (Create, List, and Update) in Airflow Versions 2.6 and below. Airflow 2.6 corresponds to Runtime Version 8, which has reached End of maintenance date and is close to reaching End of basic support (see also Basic Support).

🎟 Issue(s)

closes: #1867

🧪 Functional Testing

List the functional testing steps to confirm this feature or fix.

📸 Screenshots

Add screenshots to illustrate the validity of these changes.

Connections

AF2

❯ ./astro deployment connection create -d cmbutodaf03vx01jq3q59u9d3 --conn-id my_conn --conn-type generic
Creating connection my_conn

❯ ./astro deployment connection list -d cmbutodaf03vx01jq3q59u9d3
 CONNECTION ID     CONN TYPE     
 my_conn           generic

❯ ./astro deployment connection update -d cmbutodaf03vx01jq3q59u9d3 --conn-id my_conn --conn-type generic_2
updating connection my_conn

❯ ./astro deployment connection list -d cmbutodaf03vx01jq3q59u9d3
 CONNECTION ID     CONN TYPE     
 my_conn           generic_2

AF3

❯ ./astro deployment connection create -d cmbuofehz007l01l3m8f6i5o9 --conn-id my_conn --conn-type generic
Creating connection my_conn

❯ ./astro deployment connection list -d cmbuofehz007l01l3m8f6i5o9
 CONNECTION ID     CONN TYPE     
 my_conn           generic

❯ ./astro deployment connection update -d cmbuofehz007l01l3m8f6i5o9 --conn-id my_conn --conn-type generic_2
updating connection my_conn

❯ ./astro deployment connection list -d cmbuofehz007l01l3m8f6i5o9
 CONNECTION ID     CONN TYPE     
 my_conn           generic_2

Variables

AF2

❯ ./astro deployment airflow-variable create -d cmbutodaf03vx01jq3q59u9d3 --key foo --value bar
Creating variable foo

❯ ./astro deployment airflow-variable list -d cmbutodaf03vx01jq3q59u9d3
 KEY     DESCRIPTION     
 foo 

❯ ./astro deployment airflow-variable update -d cmbutodaf03vx01jq3q59u9d3 --key foo --value fiz
updating variable foo

AF3

❯ ./astro deployment airflow-variable create -d cmbuofehz007l01l3m8f6i5o9 --key foo --value bar
Creating variable foo

❯ ./astro deployment airflow-variable list -d cmbuofehz007l01l3m8f6i5o9
 KEY     DESCRIPTION     
 foo

❯ ./astro deployment airflow-variable update -d cmbuofehz007l01l3m8f6i5o9 --key foo --value fiz
updating variable foo

Pools

AF2

❯ ./astro deployment pool create -d cmbutodaf03vx01jq3q59u9d3 --name foo --slots 1
Creating pool foo

❯ ./astro deployment pool list -d cmbutodaf03vx01jq3q59u9d3
 NAME             SLOTS     INCLUDE DEFERRED     
 default_pool     128       false                
 foo              1         false

❯ ./astro deployment pool update -d cmbutodaf03vx01jq3q59u9d3 --name foo --slots 2 --include-deferred enable
updating pool foo

❯ ./astro deployment pool list -d cmbutodaf03vx01jq3q59u9d3
 NAME             SLOTS     INCLUDE DEFERRED     
 default_pool     128       false                
 foo              2         true

AF3

❯ ./astro deployment pool create -d cmbuofehz007l01l3m8f6i5o9 --name foo --slots 1
Creating pool foo

❯ ./astro deployment pool list -d cmbuofehz007l01l3m8f6i5o9
 NAME             SLOTS     INCLUDE DEFERRED     
 default_pool     128       false                
 foo              1         false

❯ ./astro deployment pool update -d cmbuofehz007l01l3m8f6i5o9 --name foo --slots 2 --include-deferred enable
updating pool foo

❯ ./astro deployment pool list -d cmbuofehz007l01l3m8f6i5o9
 NAME             SLOTS     INCLUDE DEFERRED     
 default_pool     128       false                
 foo              2         true

AF2.6

❯ ./astro deployment pool create -d cmbuzidah01c501m1qy7vtp2t --name foo --slots 1
Creating pool foo
Error: API error (400): {
  "detail": "{'include_deferred': ['Unknown field.']}",
  "status": 400,
  "title": "Bad Request",
  "type": "http://apache-airflow-docs.s3-website.eu-central-1.amazonaws.com/docs/apache-airflow/latest/stable-rest-api-ref.html#section/Errors/BadRequest"
}

❯ ./astro deployment pool update -d cmbuzidah01c501m1qy7vtp2t --name foo --slots 1
updating pool foo
Error: API error (400): {
  "detail": "{'include_deferred': ['Unknown field.']}",
  "status": 400,
  "title": "Bad Request",
  "type": "http://apache-airflow-docs.s3-website.eu-central-1.amazonaws.com/docs/apache-airflow/latest/stable-rest-api-ref.html#section/Errors/BadRequest"
}

📋 Checklist

  • Rebased from the main (or release if patching) branch (before testing)
  • Ran make test before taking out of draft
  • Ran make lint before taking out of draft
  • Added/updated applicable tests
  • Tested against Astro-API (if necessary).
  • Communicated to/tagged owners of respective clients potentially impacted by these changes.
  • Updated any related documentation

- update airflow client to be api version agnostic
- use metadata airflow api url for airflow client
- add include_deferred to airflow client pool schema
- add include-deferred cli flag to pool create
- add include-deferred cli flag to pool update as it is required for AF3
- show include deferred in pool listing
@feluelle feluelle force-pushed the feature/support-af3-connections-variables-pools branch from 898d618 to 8af4029 Compare June 13, 2025 17:11
Copy link
Contributor

@jaketf jaketf left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

quick work on this one thanks!

@jaketf
Copy link
Contributor

jaketf commented Jun 13, 2025

@feluelle I do think we'll need to update the docs to reflect this new support.
@neel-astro @yanmastin-astro can you remind me what the process is for staging docs changes for CLI ahead of releasing a new version?

@neel-astro
Copy link
Contributor

@neel-astro @yanmastin-astro can you remind me what the process is for staging docs changes for CLI ahead of releasing a new version?

Update the issue or PR with any details that we need to add in the docs. While creating the pre-release we should collect all the issues or PRs being added in the release and share it with Yan to help with the docs changes.
Note: give atleast one week heads up for the release process and docs changes.

Copy link
Contributor

@neel-astro neel-astro left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Left a comment, functionally looks ok otherwise

@jaketf jaketf merged commit 5a096a6 into main Jun 16, 2025
6 checks passed
@jaketf jaketf deleted the feature/support-af3-connections-variables-pools branch June 16, 2025 17:05
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Add AF 3 support for deployment object commands (create, list, update)

3 participants