Skip to content

Deprecate photon and graviton selectors in databricks_spark_version data source#2687

Merged
tanmay-db merged 1 commit intomasterfrom
spark-version-deprecate-photon-graviton
Sep 14, 2023
Merged

Deprecate photon and graviton selectors in databricks_spark_version data source#2687
tanmay-db merged 1 commit intomasterfrom
spark-version-deprecate-photon-graviton

Conversation

@alexott
Copy link
Copy Markdown
Contributor

@alexott alexott commented Sep 14, 2023

Changes

Since the introduction of the Photon & Graviton support, there were separate DBR versions exposed via APIs for each flavor:

  • DBR for Intel CPUs without Photon 13.3.x-scala2.12
  • DBR for Intel CPUs with Photon 13.3.x-photon-scala2.12
  • DBR for Graviton CPUs without Photon 13.3.x-aarch64-scala2.12
  • DBR for Graviton CPUs with Photon 13.3.x-aarch64-photon-scala2.12
  • ML Runtime ...

Since DBR 14.0, no separate DBR versions are exposed for Photon & Graviton flavors, so if you use photon and/or graviton selectors, then you won't upgrade to DBR 14+.

The migration path is the following:

  • Just specify a generic version, like, 14.0.x-scala2.12.
  • To use Photon, you need to specify runtime_engine="PHOTON" in the cluster configuration.
  • For Graviton, it's just enough to select Graviton node types, and the correct DBR will be installed automatically.

Tests

  • make test run locally
  • relevant change in docs/ folder
  • covered with integration tests in internal/acceptance
  • relevant acceptance tests are passing
  • using Go SDK

…ion` data source

Since introduction of the Photon & Graviton support, there were separate DBR versions
exposed via APIs for each flavor:

* DBR for Intel CPUs without Photon `13.3.x-scala2.12`
* DBR for Intel CPUs with Photon `13.3.x-photon-scala2.12`
* DBR for Graviton CPUs without Photon `13.3.x-aarch64-scala2.12`
* DBR for Graviton CPUs with Photon `13.3.x-aarch64-photon-scala2.12`
* ML Runtime ...

Since DBR 14.0, no separate DBR versions are exposed for Photon & Graviton flavors, so if
you use `photon` and/or `graviton` selectors, then you won't upgrade to DBR 14+.

Migration path is a following:

* Just specify a generic version, like, `14.0.x-scala2.12`.
* To use Photon, you need to specify `runtime_engine="PHOTON"` in the cluster configuration.
* For Graviton, it's just enough to select Graviton node types, and correct DBR will be installed automatically.
@alexott alexott requested review from a team as code owners September 14, 2023 15:27
@alexott alexott requested review from tanmay-db and removed request for a team September 14, 2023 15:27
@tanmay-db tanmay-db enabled auto-merge September 14, 2023 15:29
@codecov-commenter
Copy link
Copy Markdown

Codecov Report

Merging #2687 (5f27f7a) into master (69cef0e) will increase coverage by 0.00%.
Report is 3 commits behind head on master.
The diff coverage is 100.00%.

Impacted file tree graph

@@           Coverage Diff           @@
##           master    #2687   +/-   ##
=======================================
  Coverage   86.69%   86.70%           
=======================================
  Files         146      146           
  Lines       12668    12674    +6     
=======================================
+ Hits        10983    10989    +6     
  Misses       1118     1118           
  Partials      567      567           
Files Changed Coverage
clusters/data_spark_version.go 100.00%

@tanmay-db tanmay-db added this pull request to the merge queue Sep 14, 2023
Merged via the queue into master with commit 0ba6039 Sep 14, 2023
@tanmay-db tanmay-db deleted the spark-version-deprecate-photon-graviton branch September 14, 2023 15:47
@tanmay-db tanmay-db mentioned this pull request Sep 19, 2023
5 tasks
nkvuong pushed a commit that referenced this pull request Oct 3, 2023
…ion` data source (#2687)

Since introduction of the Photon & Graviton support, there were separate DBR versions
exposed via APIs for each flavor:

* DBR for Intel CPUs without Photon `13.3.x-scala2.12`
* DBR for Intel CPUs with Photon `13.3.x-photon-scala2.12`
* DBR for Graviton CPUs without Photon `13.3.x-aarch64-scala2.12`
* DBR for Graviton CPUs with Photon `13.3.x-aarch64-photon-scala2.12`
* ML Runtime ...

Since DBR 14.0, no separate DBR versions are exposed for Photon & Graviton flavors, so if
you use `photon` and/or `graviton` selectors, then you won't upgrade to DBR 14+.

Migration path is a following:

* Just specify a generic version, like, `14.0.x-scala2.12`.
* To use Photon, you need to specify `runtime_engine="PHOTON"` in the cluster configuration.
* For Graviton, it's just enough to select Graviton node types, and correct DBR will be installed automatically.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants