Skip to content

Comments

Implementing feature request #1328#1919

Merged
neel-astro merged 6 commits intomainfrom
feature-1328
Sep 17, 2025
Merged

Implementing feature request #1328#1919
neel-astro merged 6 commits intomainfrom
feature-1328

Conversation

@bmanan7
Copy link
Contributor

@bmanan7 bmanan7 commented Aug 21, 2025

Description

Describe the purpose of this pull request.

Enable astro deployment create --deployment-file to accept --wait and --verbosity. Historically, we blocked most flags when using --deployment-file to avoid configuration conflicts and ambiguity. This PR preserves that guardrail while whitelisting two operational flags that don’t alter the deployment spec:

  • --wait only affects command behavior (poll until healthy) after creation.
  • --verbosity only affects logging output.

All other flags remain disallowed alongside --deployment-file.

🎟 Issue(s)

Related #1328, #1894

🧪 Functional Testing

List the functional testing steps to confirm this feature or fix.

  1. Validated with --wait=true
./astro deployment create --wait=true --deployment-file deployment-preview-template.yaml --verbosity debug 
DEBU[0000] Available runtime versions: [11.18.0 12.0.0 11.16.0 12.3.0 11.5.0 11.6.0 11.14.0 11.12.0 12.9.0 11.13.0 12.1.1 11.4.0 11.10.0 12.7.0 12.8.0 11.3.0 11.7.0 11.15.0 11.2.0 11.11.0 11.0.0 11.10.1 11.17.0 11.1.0 12.4.0 12.6.0 13.0.0 11.8.0 12.7.1 13.1.0 11.15.1 12.5.0 12.10.0 11.19.0 12.1.0 12.2.0 11.9.0 3.0-8 3.0-1 3.0-7 3.0-9 3.0-2 3.0-3 3.0-4 3.0-5 3.0-6] 
DEBU[0000] Latest runtime version: 3.0-9                

Waiting for the deployment to become healthy…

This may take a few minutes
Deployment newdep-67 is now healthy
deployment:
.
.
  1. Validated with --wait=false
./astro deployment create --wait=false --deployment-file deployment-preview-template.yaml --verbosity debug 
DEBU[0001] Available runtime versions: [12.3.0 12.7.1 11.13.0 11.17.0 11.1.0 12.6.0 12.9.0 11.0.0 11.11.0 11.9.0 11.15.0 11.18.0 11.19.0 12.0.0 11.4.0 11.8.0 12.10.0 11.6.0 11.16.0 11.2.0 12.1.1 13.1.0 11.15.1 13.0.0 11.3.0 11.14.0 12.1.0 12.4.0 12.5.0 11.5.0 11.10.1 11.10.0 11.12.0 12.2.0 11.7.0 12.8.0 12.7.0 3.0-3 3.0-5 3.0-6 3.0-1 3.0-2 3.0-4 3.0-7 3.0-8 3.0-9] 
DEBU[0001] Latest runtime version: 3.0-9                
deployment:
    environment_variables:
        - is_secret: false
          key: TEST1
          updated_at: "2025-08-21T16:14:30.910058355Z"
          value: key1

.
.
  1. Validated without --wait flag
./astro deployment create --deployment-file deployment-preview-template.yaml --verbosity debug 
DEBU[0000] Available runtime versions: [11.0.0 11.8.0 12.5.0 12.3.0 11.7.0 11.17.0 12.0.0 12.7.0 11.14.0 11.4.0 11.10.0 11.11.0 11.6.0 12.4.0 11.10.1 11.9.0 12.10.0 11.18.0 12.7.1 12.6.0 11.5.0 12.8.0 11.1.0 11.15.1 12.2.0 11.19.0 12.1.0 11.15.0 11.16.0 12.9.0 11.12.0 13.0.0 12.1.1 11.13.0 11.2.0 11.3.0 13.1.0 3.0-1 3.0-2 3.0-4 3.0-5 3.0-7 3.0-8 3.0-9 3.0-3 3.0-6] 
DEBU[0000] Latest runtime version: 3.0-9                
deployment:
    environment_variables:
        - is_secret: false
          key: TEST1

.
.

Make test:

go test -count=1 -cover -coverprofile=coverage.txt -covermode=atomic ./...
	github.com/astronomer/astro-cli		coverage: 0.0% of statements
ok  	github.com/astronomer/astro-cli/airflow	8.511s	coverage: 84.8% of statements
	github.com/astronomer/astro-cli/airflow/mocks		coverage: 0.0% of statements
ok  	github.com/astronomer/astro-cli/airflow/runtimes	5.033s	coverage: 57.8% of statements
	github.com/astronomer/astro-cli/airflow/runtimes/mocks		coverage: 0.0% of statements
?   	github.com/astronomer/astro-cli/airflow/runtimes/types	[no test files]
?   	github.com/astronomer/astro-cli/airflow/types	[no test files]
ok  	github.com/astronomer/astro-cli/airflow-client	1.442s	coverage: 90.0% of statements
	github.com/astronomer/astro-cli/airflow-client/mocks		coverage: 0.0% of statements
ok  	github.com/astronomer/astro-cli/airflow_versions	1.186s	coverage: 96.0% of statements
	github.com/astronomer/astro-cli/astro-client-core		coverage: 0.0% of statements
	github.com/astronomer/astro-cli/astro-client-core/mocks		coverage: 0.0% of statements
	github.com/astronomer/astro-cli/astro-client-iam-core		coverage: 0.0% of statements
	github.com/astronomer/astro-cli/astro-client-iam-core/mocks		coverage: 0.0% of statements
	github.com/astronomer/astro-cli/astro-client-platform-core		coverage: 0.0% of statements
	github.com/astronomer/astro-cli/astro-client-platform-core/mocks		coverage: 0.0% of statements
ok  	github.com/astronomer/astro-cli/cloud/auth	8.412s	coverage: 87.8% of statements
ok  	github.com/astronomer/astro-cli/cloud/deploy	9.676s	coverage: 78.3% of statements
ok  	github.com/astronomer/astro-cli/cloud/deployment	11.103s	coverage: 89.5% of statements
ok  	github.com/astronomer/astro-cli/cloud/deployment/fromfile	6.329s	coverage: 91.3% of statements
ok  	github.com/astronomer/astro-cli/cloud/deployment/inspect	1.971s	coverage: 91.3% of statements
ok  	github.com/astronomer/astro-cli/cloud/deployment/workerqueue	2.545s	coverage: 94.3% of statements
ok  	github.com/astronomer/astro-cli/cloud/environment	2.774s	coverage: 92.6% of statements
ok  	github.com/astronomer/astro-cli/cloud/organization	3.079s	coverage: 89.6% of statements
ok  	github.com/astronomer/astro-cli/cloud/role	3.338s	coverage: 100.0% of statements
ok  	github.com/astronomer/astro-cli/cloud/team	3.686s	coverage: 91.0% of statements
ok  	github.com/astronomer/astro-cli/cloud/user	3.348s	coverage: 94.6% of statements
ok  	github.com/astronomer/astro-cli/cloud/workspace	3.405s	coverage: 90.3% of statements
ok  	github.com/astronomer/astro-cli/cloud/workspace-token	2.524s	coverage: 88.5% of statements
ok  	github.com/astronomer/astro-cli/cmd	3.769s	coverage: 89.9% of statements
ok  	github.com/astronomer/astro-cli/cmd/cloud	3.833s	coverage: 90.8% of statements
ok  	github.com/astronomer/astro-cli/cmd/registry	2.913s	coverage: 69.6% of statements
ok  	github.com/astronomer/astro-cli/cmd/software	1.753s	coverage: 91.7% of statements
ok  	github.com/astronomer/astro-cli/cmd/utils	1.774s	coverage: 100.0% of statements
ok  	github.com/astronomer/astro-cli/config	2.432s	coverage: 84.5% of statements
ok  	github.com/astronomer/astro-cli/context	2.132s	coverage: 80.5% of statements
ok  	github.com/astronomer/astro-cli/docker	1.631s	coverage: 74.4% of statements
ok  	github.com/astronomer/astro-cli/houston	2.600s	coverage: 90.1% of statements
	github.com/astronomer/astro-cli/houston/mocks		coverage: 0.0% of statements
ok  	github.com/astronomer/astro-cli/pkg/ansi	1.121s	coverage: 81.0% of statements
ok  	github.com/astronomer/astro-cli/pkg/azure	1.376s	coverage: 11.1% of statements
	github.com/astronomer/astro-cli/pkg/azure/mocks		coverage: 0.0% of statements
ok  	github.com/astronomer/astro-cli/pkg/domainutil	1.525s	coverage: 100.0% of statements
ok  	github.com/astronomer/astro-cli/pkg/fileutil	6.635s	coverage: 86.7% of statements
ok  	github.com/astronomer/astro-cli/pkg/git	1.654s	coverage: 36.0% of statements
	github.com/astronomer/astro-cli/pkg/httputil		coverage: 0.0% of statements
ok  	github.com/astronomer/astro-cli/pkg/input	1.551s	coverage: 90.9% of statements
	github.com/astronomer/astro-cli/pkg/logger		coverage: 0.0% of statements
ok  	github.com/astronomer/astro-cli/pkg/printutil	1.670s	coverage: 93.1% of statements
	github.com/astronomer/astro-cli/pkg/spinner		coverage: 0.0% of statements
	github.com/astronomer/astro-cli/pkg/testing		coverage: 0.0% of statements
ok  	github.com/astronomer/astro-cli/pkg/util	1.695s	coverage: 95.3% of statements
ok  	github.com/astronomer/astro-cli/settings	1.888s	coverage: 86.2% of statements
ok  	github.com/astronomer/astro-cli/software/auth	1.107s	coverage: 89.9% of statements
ok  	github.com/astronomer/astro-cli/software/deploy	1.842s	coverage: 90.2% of statements
ok  	github.com/astronomer/astro-cli/software/deployment	1.465s	coverage: 92.4% of statements
ok  	github.com/astronomer/astro-cli/software/service_account	1.143s	coverage: 100.0% of statements
ok  	github.com/astronomer/astro-cli/software/teams	2.106s	coverage: 94.2% of statements
ok  	github.com/astronomer/astro-cli/software/user	2.398s	coverage: 94.1% of statements
	github.com/astronomer/astro-cli/software/utils		coverage: 0.0% of statements
ok  	github.com/astronomer/astro-cli/software/workspace	2.651s	coverage: 96.2% of statements
ok  	github.com/astronomer/astro-cli/version	2.747s	coverage: 12.9% of statements
manan.bhatt@192 astro-cli % go test -v -count=1 ./cmd/cloud -run '^TestDeploymentCreate'                                      
=== RUN   TestDeploymentCreate
=== RUN   TestDeploymentCreate/creates_a_deployment_when_dag-deploy_is_disabled
Current Workspace: test-workspace

 NAME     NAMESPACE     CLUSTER          CLOUD PROVIDER     REGION     DEPLOYMENT ID     RUNTIME VERSION          DAG DEPLOY ENABLED     CI-CD ENFORCEMENT     DEPLOYMENT TYPE     
                        test-cluster                                   test-id            (based on Airflow )     false                  false                 HYBRID              

 Successfully created Deployment: 
 Deployment can be accessed at the following URLs 

 Deployment Dashboard: localhost:5000/workspace-id/deployments/test-id
 Airflow Dashboard: 
=== RUN   TestDeploymentCreate/creates_a_deployment_when_dag_deploy_is_enabled
Current Workspace: test-workspace

 NAME     NAMESPACE     CLUSTER          CLOUD PROVIDER     REGION     DEPLOYMENT ID     RUNTIME VERSION          DAG DEPLOY ENABLED     CI-CD ENFORCEMENT     DEPLOYMENT TYPE     
                        test-cluster                                   test-id            (based on Airflow )     false                  false                 HYBRID              

 Successfully created Deployment: 
 Deployment can be accessed at the following URLs 

 Deployment Dashboard: localhost:5000/workspace-id/deployments/test-id
 Airflow Dashboard: 
=== RUN   TestDeploymentCreate/creates_a_deployment_when_executor_is_specified
Current Workspace: test-workspace

 NAME     NAMESPACE     CLUSTER          CLOUD PROVIDER     REGION     DEPLOYMENT ID     RUNTIME VERSION          DAG DEPLOY ENABLED     CI-CD ENFORCEMENT     DEPLOYMENT TYPE     
                        test-cluster                                   test-id            (based on Airflow )     false                  false                 HYBRID              

 Successfully created Deployment: 
 Deployment can be accessed at the following URLs 

 Deployment Dashboard: localhost:5000/workspace-id/deployments/test-id
 Airflow Dashboard: 
=== RUN   TestDeploymentCreate/creates_a_deployment_with_default_executor
Current Workspace: test-workspace

 NAME     NAMESPACE     CLUSTER          CLOUD PROVIDER     REGION     DEPLOYMENT ID     RUNTIME VERSION          DAG DEPLOY ENABLED     CI-CD ENFORCEMENT     DEPLOYMENT TYPE     
                        test-cluster                                   test-id            (based on Airflow )     false                  false                 HYBRID              

 Successfully created Deployment: 
 Deployment can be accessed at the following URLs 

 Deployment Dashboard: localhost:5000/workspace-id/deployments/test-id
 Airflow Dashboard: 
=== RUN   TestDeploymentCreate/returns_an_error_if_dag-deploy_flag_has_an_incorrect_value
Error: Invalid --dag-deploy value)
=== RUN   TestDeploymentCreate/returns_an_error_if_cluster-type_flag_has_an_incorrect_value
Error: unknown flag: --cluster-type
=== RUN   TestDeploymentCreate/returns_an_error_if_type_flag_has_an_incorrect_value
Error: unknown flag: --type
=== RUN   TestDeploymentCreate/returns_an_error_if_cicd-enforcement_flag_has_an_incorrect_value
Error: Invalid --cicd-enforcement value
=== RUN   TestDeploymentCreate/returns_an_error_if_executor_has_an_incorrect_value
Error: KubeExecutor is not a valid executor for runtime version 4.2.5 deployment type standard
=== RUN   TestDeploymentCreate/returns_an_error_if_remote-execution-enabled_flag_is_set_but_org_is_not_hosted
Error: unknown flag: --remote-execution-enabled
=== RUN   TestDeploymentCreate/creates_a_deployment_from_file
=== RUN   TestDeploymentCreate/returns_an_error_if_creating_a_deployment_from_file_fails
Error: open test-file-name.json: no such file or directory
=== RUN   TestDeploymentCreate/returns_an_error_if_from-file_is_specified_with_any_other_flags
Error: --deployment-file can not be used with other arguments
=== RUN   TestDeploymentCreate/creates_a_deployment_from_file_when_supported_flags_are_set

Waiting for the deployment to become healthy…

This may take a few minutes
Deployment test-deployment-label is now healthy
=== RUN   TestDeploymentCreate/returns_an_error_if_from-file_is_specified_with_supported_and_unsupported_flags
Error: --deployment-file can not be used with other arguments
=== RUN   TestDeploymentCreate/creates_a_deployment_with_cloud_provider_and_region
Current Workspace: test-workspace

 NAME     NAMESPACE     CLUSTER          CLOUD PROVIDER     REGION          DEPLOYMENT ID     RUNTIME VERSION          DAG DEPLOY ENABLED     CI-CD ENFORCEMENT     DEPLOYMENT TYPE     
                        test-cluster     gcp                us-central1     test-id            (based on Airflow )     false                  false                 HYBRID              

 Successfully created Deployment: 
 Deployment can be accessed at the following URLs 

 Deployment Dashboard: localhost:5000/workspace-id/deployments/test-id
 Airflow Dashboard: 
=== RUN   TestDeploymentCreate/returns_an_error_with_incorrect_high-availability_value
Error: Invalid --high-availability value
=== RUN   TestDeploymentCreate/returns_an_error_with_incorrect_development-mode_value
Error: Invalid --development-mode value
=== RUN   TestDeploymentCreate/returns_an_error_if_cloud_provider_is_not_valid
Error: ibm is not a valid cloud provider. It can only be gcp, azure or aws
=== RUN   TestDeploymentCreate/creates_a_hosted_dedicated_deployment
Current Workspace: test-workspace


Please select a Cluster for your Deployment:
 #     CLUSTER NAME       CLOUD PROVIDER     CLUSTER ID            
 1     test-cluster                          test-cluster-id       
 2     test-cluster-1                        test-cluster-id-1     

>  NAME     NAMESPACE     CLUSTER          CLOUD PROVIDER     REGION     DEPLOYMENT ID     RUNTIME VERSION          DAG DEPLOY ENABLED     CI-CD ENFORCEMENT     DEPLOYMENT TYPE     
                        test-cluster     azure                         test-id            (based on Airflow )     false                  false                 HYBRID              

 Successfully created Deployment: 
 Deployment can be accessed at the following URLs 

 Deployment Dashboard: localhost:5000/workspace-id/deployments/test-id
 Airflow Dashboard: 
=== RUN   TestDeploymentCreate/returns_an_error_if_incorrect_cluster_type_is_passed_for_a_hosted_dedicated_deployment
Error: Invalid --type value
=== RUN   TestDeploymentCreate/creates_an_extra_large_deployment
Current Workspace: test-workspace


Please select a Cluster for your Deployment:
 #     CLUSTER NAME       CLOUD PROVIDER     CLUSTER ID            
 1     test-cluster                          test-cluster-id       
 2     test-cluster-1                        test-cluster-id-1     

>  NAME     NAMESPACE     CLUSTER          CLOUD PROVIDER     REGION     DEPLOYMENT ID     RUNTIME VERSION          DAG DEPLOY ENABLED     CI-CD ENFORCEMENT     DEPLOYMENT TYPE     
                        test-cluster     azure                         test-id            (based on Airflow )     false                  false                 HYBRID              

 Successfully created Deployment: 
 Deployment can be accessed at the following URLs 

 Deployment Dashboard: localhost:5000/workspace-id/deployments/test-id
 Airflow Dashboard: 
=== RUN   TestDeploymentCreate/creates_a_hosted_deployment_with_workload_identity
Current Workspace: test-workspace

 NAME     NAMESPACE     CLUSTER          CLOUD PROVIDER     REGION        DEPLOYMENT ID     RUNTIME VERSION          DAG DEPLOY ENABLED     CI-CD ENFORCEMENT     DEPLOYMENT TYPE     
                        test-cluster     aws                us-west-2     test-id            (based on Airflow )     false                  false                 HYBRID              

 Successfully created Deployment: 
 Deployment can be accessed at the following URLs 

 Deployment Dashboard: localhost:5000/workspace-id/deployments/test-id
 Airflow Dashboard: 
--- PASS: TestDeploymentCreate (1.04s)
    --- PASS: TestDeploymentCreate/creates_a_deployment_when_dag-deploy_is_disabled (0.00s)
    --- PASS: TestDeploymentCreate/creates_a_deployment_when_dag_deploy_is_enabled (0.00s)
    --- PASS: TestDeploymentCreate/creates_a_deployment_when_executor_is_specified (0.00s)
    --- PASS: TestDeploymentCreate/creates_a_deployment_with_default_executor (0.00s)
    --- PASS: TestDeploymentCreate/returns_an_error_if_dag-deploy_flag_has_an_incorrect_value (0.00s)
    --- PASS: TestDeploymentCreate/returns_an_error_if_cluster-type_flag_has_an_incorrect_value (0.00s)
    --- PASS: TestDeploymentCreate/returns_an_error_if_type_flag_has_an_incorrect_value (0.00s)
    --- PASS: TestDeploymentCreate/returns_an_error_if_cicd-enforcement_flag_has_an_incorrect_value (0.00s)
    --- PASS: TestDeploymentCreate/returns_an_error_if_executor_has_an_incorrect_value (0.00s)
    --- PASS: TestDeploymentCreate/returns_an_error_if_remote-execution-enabled_flag_is_set_but_org_is_not_hosted (0.00s)
    --- PASS: TestDeploymentCreate/creates_a_deployment_from_file (0.00s)
    --- PASS: TestDeploymentCreate/returns_an_error_if_creating_a_deployment_from_file_fails (0.00s)
    --- PASS: TestDeploymentCreate/returns_an_error_if_from-file_is_specified_with_any_other_flags (0.00s)
    --- PASS: TestDeploymentCreate/creates_a_deployment_from_file_when_supported_flags_are_set (1.01s)
    --- PASS: TestDeploymentCreate/returns_an_error_if_from-file_is_specified_with_supported_and_unsupported_flags (0.00s)
    --- PASS: TestDeploymentCreate/creates_a_deployment_with_cloud_provider_and_region (0.00s)
    --- PASS: TestDeploymentCreate/returns_an_error_with_incorrect_high-availability_value (0.00s)
    --- PASS: TestDeploymentCreate/returns_an_error_with_incorrect_development-mode_value (0.00s)
    --- PASS: TestDeploymentCreate/returns_an_error_if_cloud_provider_is_not_valid (0.00s)
    --- PASS: TestDeploymentCreate/creates_a_hosted_dedicated_deployment (0.00s)
    --- PASS: TestDeploymentCreate/returns_an_error_if_incorrect_cluster_type_is_passed_for_a_hosted_dedicated_deployment (0.00s)
    --- PASS: TestDeploymentCreate/creates_an_extra_large_deployment (0.00s)
    --- PASS: TestDeploymentCreate/creates_a_hosted_deployment_with_workload_identity (0.00s)
PASS
ok  	github.com/astronomer/astro-cli/cmd/cloud	1.972s

📸 Screenshots

Add screenshots to illustrate the validity of these changes.

image image

📋 Checklist

  • Rebased from the main (or release if patching) branch (before testing)
  • Ran make test before taking out of draft
  • Ran make lint before taking out of draft
  • Added/updated applicable tests
  • Tested against Astro-API (if necessary).
  • Tested against Houston-API and Astronomer (if necessary).
  • Communicated to/tagged owners of respective clients potentially impacted by these changes.
  • [] Updated any related documentation

Copy link
Contributor

@neel-astro neel-astro left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks for taking it up ❤️ , the change seems ok, but can we please add unit tests for all the newly added logic

@bmanan7 bmanan7 marked this pull request as draft September 1, 2025 03:32
@coveralls-official
Copy link

coveralls-official bot commented Sep 1, 2025

Pull Request Test Coverage Report for Build 4912ce8e-d882-467c-92ec-0e8aad0263f2

Details

  • 33 of 35 (94.29%) changed or added relevant lines in 3 files are covered.
  • 2 unchanged lines in 1 file lost coverage.
  • Overall coverage increased (+0.02%) to 38.151%

Changes Missing Coverage Covered Lines Changed/Added Lines %
cloud/deployment/fromfile/fromfile.go 18 20 90.0%
Files with Coverage Reduction New Missed Lines %
docker/docker.go 2 46.81%
Totals Coverage Status
Change from base Build a0f74993-783a-426a-8ffd-30b07b242e98: 0.02%
Covered Lines: 23768
Relevant Lines: 62300

💛 - Coveralls

@bmanan7
Copy link
Contributor Author

bmanan7 commented Sep 1, 2025

Thanks for taking it up ❤️ , the change seems ok, but can we please add unit tests for all the newly added logic

Added TestWaitForStatus to cover the new waitForStatus feature; overall coverage improved and core behavior is fully tested. Thanks for the initial feedback!

These live under TestWaitForStatus in fromfile_test.go

  • Verified four scenarios:
    • Create + wait=true → polls until HEALTHY
    • Create + wait=false → no polling
    • Create + wait=true but never healthy → returns deployment.ErrTimedOut
    • Create + wait=true with API error → propagates error
  • Polling is explicitly guarded by waitForStatus and only enabled for create; update never polls.
  • Polling tunables are set to minimal values in tests to keep them fast and deterministic.

@bmanan7 bmanan7 self-assigned this Sep 1, 2025
@bmanan7 bmanan7 marked this pull request as ready for review September 1, 2025 08:14
@bmanan7 bmanan7 requested a review from neel-astro September 1, 2025 08:15
Copy link
Contributor

@neel-astro neel-astro left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks for adding this; it was a long-awaited feature. I am approving the PR to not be blocked on review, but I left a few tiny comments to be addressed before merging

@neel-astro neel-astro merged commit 8e9bb4b into main Sep 17, 2025
5 of 6 checks passed
@neel-astro neel-astro deleted the feature-1328 branch September 17, 2025 12:21
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants