Skip to content

Comments

Enable deployment worker queue cmd for AF3#1887

Merged
jaketf merged 4 commits intomainfrom
feature/deployment-worker-queue-af3
Jul 1, 2025
Merged

Enable deployment worker queue cmd for AF3#1887
jaketf merged 4 commits intomainfrom
feature/deployment-worker-queue-af3

Conversation

@feluelle
Copy link
Member

@feluelle feluelle commented Jul 1, 2025 •
edited
Loading

Description

Describe the purpose of this pull request.

Removes the AF3 gates on astro deployment worker-queue to enable the programmatic support

depends on #1886 as it uses deployment update to update worker queues.

🎟 Issue(s)

Related #XXX

🧪 Functional Testing

List the functional testing steps to confirm this feature or fix.

📸 Screenshots

Add screenshots to illustrate the validity of these changes.

Inspect deployment

❯ ~/Projects/astro-cli/astro deployment inspect
Select a Deployment
 #     DEPLOYMENT NAME     RELEASE NAME         DEPLOYMENT ID                 DAG DEPLOY ENABLED     
 1     test                modern-beam-2288     cmckmn35w0er001l4fw8v8136     true                   

> 1
deployment:
    ...
    worker_queues:
        - name: asd
          min_worker_count: 0
          worker_type: A5
          pod_cpu: "1"
          pod_ram: 2Gi
        - name: default
          min_worker_count: 0
          worker_type: A5
          pod_cpu: "1"
          pod_ram: 2Gi

Create worker queue

❯ ~/Projects/astro-cli/astro deployment worker-queue create
Select a Deployment
 #     DEPLOYMENT NAME     RELEASE NAME         DEPLOYMENT ID                 DAG DEPLOY ENABLED     
 1     test                modern-beam-2288     cmckmn35w0er001l4fw8v8136     true                   

> 1
Enter a name for the worker queue
> foo
No worker type was specified. Select the worker type to use
 #     WORKER TYPE     CPU         Memory     
 1     A5              1 vCPU      2Gi        
 2     A10             2 vCPU      4Gi        
 3     A20             4 vCPU      8Gi        
 4     A40             8 vCPU      16Gi       
 5     A60             12 vCPU     24Gi       
 6     A120            24 vCPU     48Gi       
 7     A160            32 vCPU     64Gi       

> 2
worker queue foo for test in cm00phuvy06h201o3m1vme6i2 workspace created

Inspect deployment

❯ ~/Projects/astro-cli/astro deployment inspect
Select a Deployment
 #     DEPLOYMENT NAME     RELEASE NAME         DEPLOYMENT ID                 DAG DEPLOY ENABLED     
 1     test                modern-beam-2288     cmckmn35w0er001l4fw8v8136     true                   

> 1
deployment:
    ...
    worker_queues:
        - name: foo
          min_worker_count: 0
          worker_type: A10
          pod_cpu: "2"
          pod_ram: 4Gi
        - name: asd
          min_worker_count: 0
          worker_type: A5
          pod_cpu: "1"
          pod_ram: 2Gi
        - name: default
          min_worker_count: 0
          worker_type: A5
          pod_cpu: "1"
          pod_ram: 2Gi

Update worker queue

❯ ~/Projects/astro-cli/astro deployment worker-queue update --worker-type A5
Select a Deployment
 #     DEPLOYMENT NAME     RELEASE NAME         DEPLOYMENT ID                 DAG DEPLOY ENABLED     
 1     test                modern-beam-2288     cmckmn35w0er001l4fw8v8136     true                   

> 1

 #     WORKER QUEUE     ISDEFAULT     ID                            
 1     asd              false         cmckqhruh034c01n3ol3uu444     
 2     default          true          cmckqg3dz031b01n3ora29q49     
 3     foo              false         cmckrdem40a0r01ltdcbp1bki     

> 3

Are you sure you want to update the foo worker queue? If there are any tasks in your DAGs assigned to this worker queue, the tasks might get stuck in a queued state and fail to execute (y/n) y
worker queue foo for test in cm00phuvy06h201o3m1vme6i2 workspace updated

Inspect deployment

❯ ~/Projects/astro-cli/astro deployment inspect
Select a Deployment
 #     DEPLOYMENT NAME     RELEASE NAME         DEPLOYMENT ID                 DAG DEPLOY ENABLED     
 1     test                modern-beam-2288     cmckmn35w0er001l4fw8v8136     true                   

> 1
deployment:
    ...
    worker_queues:
        - name: default
          min_worker_count: 0
          worker_type: A5
          pod_cpu: "1"
          pod_ram: 2Gi
        - name: foo
          min_worker_count: 0
          worker_type: A5
          pod_cpu: "1"
          pod_ram: 2Gi
        - name: asd
          min_worker_count: 0
          worker_type: A5
          pod_cpu: "1"
          pod_ram: 2Gi

📋 Checklist

  • Rebased from the main (or release if patching) branch (before testing)
  • Ran make test before taking out of draft
  • Ran make lint before taking out of draft
  • Added/updated applicable tests
  • Tested against Astro-API (if necessary).
  • Tested against Houston-API and Astronomer (if necessary).
  • Communicated to/tagged owners of respective clients potentially impacted by these changes.
  • Updated any related documentation

@feluelle feluelle requested a review from a team July 1, 2025 16:31
@feluelle feluelle self-assigned this Jul 1, 2025
@feluelle feluelle force-pushed the feature/deployment-worker-queue-af3 branch from 5a0f534 to ccd459f Compare July 1, 2025 16:39
@feluelle feluelle marked this pull request as draft July 1, 2025 17:00
@jaketf jaketf marked this pull request as ready for review July 1, 2025 18:18
@jaketf jaketf merged commit 6e3b986 into main Jul 1, 2025
4 of 5 checks passed
@jaketf jaketf deleted the feature/deployment-worker-queue-af3 branch July 1, 2025 18:26
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@jaketf jaketf jaketf approved these changes

@neel-astro neel-astro Awaiting requested review from neel-astro

@jeremybeard jeremybeard Awaiting requested review from jeremybeard

@kushalmalani kushalmalani Awaiting requested review from kushalmalani

Assignees

@feluelle feluelle

Labels

None yet

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

2 participants

@feluelle @jaketf