Skip to content

Retry API calls until context timeout#3537

Merged
mgyucht merged 1 commit intomainfrom
retry-until-timeout
May 3, 2024
Merged

Retry API calls until context timeout#3537
mgyucht merged 1 commit intomainfrom
retry-until-timeout

Conversation

@mgyucht
Copy link
Copy Markdown
Contributor

@mgyucht mgyucht commented May 2, 2024

Changes

The SDK currently sets a default retry timeout of 5 minutes for all API requests. This is convenient for users who call APIs with contexts with no timeout or that can't be cancelled, like context.Background(). However, in Terraform, the context timeout is determined by a combination of instance-level timeouts, resource-level timeouts, and a default timeout of 20m.

This PR changes the provider to use an indefinite retry timeout for all API requests. This allows API calls to be retried for the entire length of the resource creation operation, e.g. in case there is a low rate limit for an API and the resources in question cannot all be created within 5 minutes.

Tests

  • make test run locally
  • relevant change in docs/ folder
  • covered with integration tests in internal/acceptance
  • relevant acceptance tests are passing
  • using Go SDK

@mgyucht mgyucht requested review from a team as code owners May 2, 2024 15:40
@mgyucht mgyucht requested review from hectorcast-db and removed request for a team May 2, 2024 15:40
@codecov-commenter
Copy link
Copy Markdown

Codecov Report

All modified and coverable lines are covered by tests ✅

Project coverage is 82.70%. Comparing base (4927dc0) to head (49ebe8f).

Additional details and impacted files

Impacted file tree graph

@@            Coverage Diff             @@
##             main    #3537      +/-   ##
==========================================
+ Coverage   82.69%   82.70%   +0.01%     
==========================================
  Files         183      183              
  Lines       18826    18829       +3     
==========================================
+ Hits        15568    15573       +5     
+ Misses       2341     2340       -1     
+ Partials      917      916       -1     
Files Coverage Δ
provider/provider.go 93.83% <100.00%> (+0.08%) ⬆️

... and 1 file with indirect coverage changes

@mgyucht mgyucht added this pull request to the merge queue May 3, 2024
Merged via the queue into main with commit def3a22 May 3, 2024
@mgyucht mgyucht deleted the retry-until-timeout branch May 3, 2024 07:39
mgyucht added a commit that referenced this pull request May 3, 2024
### New Features and Improvements
* Allow changes to `display_name` for `databricks_service_principal` ([#3523](#3523)).
* Modify `databricks_repo` to support Git folders in the workspace ([#3447](#3447)).
* Retry requests in the Databricks Go SDK until context timeout ([#3537](#3537)).
* Set ForceSendFields for boolean/integer values explicitly set to false/0 ([#3385](#3385)).

### Bug fixes
* Added `ConflictsWith` to legacy tasks in `databricks_job` to avoid errors when job parameters are used ([#3500](#3500)).
* Fixed spn secret permanent diff ([#3525](#3525)).
* Introduced `common.NoClientData` helper function and use it in `databricks_aws_crossaccount_policy` data source to allow its use with workspace- or account-level providers ([#3508](#3508)).
* Set `id` attribute of `databricks_metastore` data source ([#3503](#3503)).

### Documentation Changes
* Added documentation for compute options for tasks in `databricks_job` resource ([#3504](#3504)).
* Added warning about depends_on order ([#3489](#3489)).
* Fixed various doc typos ([#3512](#3512)).
* Fixed `databricks_current_metastore` data source names in docs examples ([#3515](#3515)).
* Improved Job documentation structure ([#3511](#3511)).
* Removed confusing mention of cloud types for `databricks_clusters` and `databricks_sql_warehouses` data sources ([#3518](#3518)).

### Exporter
* Added UC support for DLT pipelines ([#3494](#3494)).
* Ignored instance pools with the empty name ([#3522](#3522)).

### Internal Changes
* Added DATABRICKS_USER_AGENT_EXTRA ([#3520](#3520)).
* Adjusted list of supported platforms in `godownloader-databricks-provider.sh` ([#3527](#3527)).
* Switched to Go SDK for `databricks_directory` resource and data source ([#3509](#3509)).
* Fixed codecov for repository ([#3530](#3530)).

Dependency updates:

 * Bump databricks-sdk-go to v0.40.0 ([#3534](#3534)).
@mgyucht mgyucht mentioned this pull request May 3, 2024
github-merge-queue bot pushed a commit that referenced this pull request May 3, 2024
* Release v1.42.0

### New Features and Improvements
* Allow changes to `display_name` for `databricks_service_principal` ([#3523](#3523)).
* Modify `databricks_repo` to support Git folders in the workspace ([#3447](#3447)).
* Retry requests in the Databricks Go SDK until context timeout ([#3537](#3537)).
* Set ForceSendFields for boolean/integer values explicitly set to false/0 ([#3385](#3385)).

### Bug fixes
* Added `ConflictsWith` to legacy tasks in `databricks_job` to avoid errors when job parameters are used ([#3500](#3500)).
* Fixed spn secret permanent diff ([#3525](#3525)).
* Introduced `common.NoClientData` helper function and use it in `databricks_aws_crossaccount_policy` data source to allow its use with workspace- or account-level providers ([#3508](#3508)).
* Set `id` attribute of `databricks_metastore` data source ([#3503](#3503)).

### Documentation Changes
* Added documentation for compute options for tasks in `databricks_job` resource ([#3504](#3504)).
* Added warning about depends_on order ([#3489](#3489)).
* Fixed various doc typos ([#3512](#3512)).
* Fixed `databricks_current_metastore` data source names in docs examples ([#3515](#3515)).
* Improved Job documentation structure ([#3511](#3511)).
* Removed confusing mention of cloud types for `databricks_clusters` and `databricks_sql_warehouses` data sources ([#3518](#3518)).

### Exporter
* Added UC support for DLT pipelines ([#3494](#3494)).
* Ignored instance pools with the empty name ([#3522](#3522)).

### Internal Changes
* Added DATABRICKS_USER_AGENT_EXTRA ([#3520](#3520)).
* Adjusted list of supported platforms in `godownloader-databricks-provider.sh` ([#3527](#3527)).
* Switched to Go SDK for `databricks_directory` resource and data source ([#3509](#3509)).
* Fixed codecov for repository ([#3530](#3530)).

Dependency updates:

 * Bump databricks-sdk-go to v0.40.0 ([#3534](#3534)).

* Changelog
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants