Skip to content

Release v0.3.1#87

Merged
nfx merged 1 commit intomainfrom
prepare/0.3.1
Apr 2, 2024
Merged

Release v0.3.1#87
nfx merged 1 commit intomainfrom
prepare/0.3.1

Conversation

@nfx
Copy link
Copy Markdown
Collaborator

@nfx nfx commented Apr 2, 2024

  • Check UCX and LSQL for backwards compatibility (#78). In this release, we introduce a new GitHub Actions workflow, downstreams.yml, which automates unit testing for downstream projects upon changes made to the upstream project. The workflow runs on pull requests, merge groups, and pushes to the main branch and sets permissions for id-token, contents, and pull-requests. It includes a compatibility job that runs on Ubuntu, checks out the code, sets up Python, installs the toolchain, and accepts downstream projects using the databrickslabs/sandbox/downstreams action. The job matrix includes two downstream projects, ucx and remorph, and uses the build cache to speed up the pip install step. This feature ensures that changes to the upstream project do not break compatibility with downstream projects, maintaining a stable and reliable library for software engineers.
  • Fixed Builder object has no attribute sdk_config error (#86). In this release, we've resolved a Builder object has no attribute sdk_config error that occurred when initializing a Spark session using the DatabricksSession.builder method. The issue was caused by using dot notation to access the sdk_config attribute, which is incorrect. This has been updated to the correct syntax of sdkConfig. This change enables successful creation of the Spark session, preventing the error from recurring. The DatabricksSession class and its methods, such as getOrCreate, continue to be used for interacting with Databricks clusters and workspaces, while the WorkspaceClient class manages Databricks resources within a workspace.

Dependency updates:

  • Bump codecov/codecov-action from 1 to 4 (#84).
  • Bump actions/setup-python from 4 to 5 (#83).
  • Bump actions/checkout from 2.5.0 to 4.1.2 (#81).
  • Bump softprops/action-gh-release from 1 to 2 (#80).

@nfx
Release v0.3.1
63adb9f 
* Check UCX and LSQL for backwards compatibility ([#78](#78)). In this release, we introduce a new GitHub Actions workflow, downstreams.yml, which automates unit testing for downstream projects upon changes made to the upstream project. The workflow runs on pull requests, merge groups, and pushes to the main branch and sets permissions for id-token, contents, and pull-requests. It includes a compatibility job that runs on Ubuntu, checks out the code, sets up Python, installs the toolchain, and accepts downstream projects using the databrickslabs/sandbox/downstreams action. The job matrix includes two downstream projects, ucx and remorph, and uses the build cache to speed up the pip install step. This feature ensures that changes to the upstream project do not break compatibility with downstream projects, maintaining a stable and reliable library for software engineers.
* Fixed `Builder` object has no attribute `sdk_config` error ([#86](#86)). In this release, we've resolved a `Builder` object has no attribute `sdk_config` error that occurred when initializing a Spark session using the `DatabricksSession.builder` method. The issue was caused by using dot notation to access the `sdk_config` attribute, which is incorrect. This has been updated to the correct syntax of `sdkConfig`. This change enables successful creation of the Spark session, preventing the error from recurring. The `DatabricksSession` class and its methods, such as `getOrCreate`, continue to be used for interacting with Databricks clusters and workspaces, while the `WorkspaceClient` class manages Databricks resources within a workspace.

Dependency updates:

 * Bump codecov/codecov-action from 1 to 4 ([#84](#84)).
 * Bump actions/setup-python from 4 to 5 ([#83](#83)).
 * Bump actions/checkout from 2.5.0 to 4.1.2 ([#81](#81)).
 * Bump softprops/action-gh-release from 1 to 2 ([#80](#80)).
@github-actions
Copy link
Copy Markdown

github-actions bot commented Apr 2, 2024

This PR breaks backwards compatibility for databrickslabs/ucx downstream. See build logs for more details.

Running from downstreams #19

@github-actions
Copy link
Copy Markdown

github-actions bot commented Apr 2, 2024

❌ 17/18 passed, 1 failed, 2 skipped, 10m49s total

❌ test_overwrite: databricks.sdk.errors.platform.BadRequest: [INSUFFICIENT_PERMISSIONS] Insufficient privileges: (692ms) 
databricks.sdk.errors.platform.BadRequest: [INSUFFICIENT_PERMISSIONS] Insufficient privileges:
User does not have permission CREATE,USAGE on database `default`.
16:06 DEBUG [databricks.sdk] Loaded from environment
16:06 DEBUG [databricks.sdk] Ignoring pat auth, because metadata-service is preferred
16:06 DEBUG [databricks.sdk] Ignoring basic auth, because metadata-service is preferred
16:06 DEBUG [databricks.sdk] Attempting to configure auth: metadata-service
16:06 INFO [databricks.sdk] Using Databricks Metadata Service authentication
[gw8] linux -- Python 3.10.14 /home/runner/work/lsql/lsql/.venv/bin/python
16:06 DEBUG [databricks.sdk] Loaded from environment
16:06 DEBUG [databricks.sdk] Ignoring pat auth, because metadata-service is preferred
16:06 DEBUG [databricks.sdk] Ignoring basic auth, because metadata-service is preferred
16:06 DEBUG [databricks.sdk] Attempting to configure auth: metadata-service
16:06 INFO [databricks.sdk] Using Databricks Metadata Service authentication
16:06 DEBUG [databricks.labs.lsql.backends] [api][execute] CREATE TABLE IF NOT EXISTS hive_metastore.default.foo (first STRING NOT NULL, second BOOLEAN NOT... (18 more bytes)
16:06 DEBUG [databricks.labs.lsql.core] Executing SQL statement: CREATE TABLE IF NOT EXISTS hive_metastore.default.foo (first STRING NOT NULL, second BOOLEAN NOT NULL) USING DELTA
16:06 DEBUG [databricks.sdk] POST /api/2.0/sql/statements/
> {
>   "format": "JSON_ARRAY",
>   "statement": "CREATE TABLE IF NOT EXISTS hive_metastore.default.foo (first STRING NOT NULL, second BOOLEAN NOT... (18 more bytes)",
>   "warehouse_id": "TEST_DEFAULT_WAREHOUSE_ID"
> }
< 200 OK
< {
<   "statement_id": "01eef10b-016f-1931-9a1e-8029529a2db2",
<   "status": {
<     "error": {
<       "error_code": "BAD_REQUEST",
<       "message": "[INSUFFICIENT_PERMISSIONS] Insufficient privileges:\nUser does not have permission CREATE,USAGE o... (21 more bytes)"
<     },
<     "state": "FAILED"
<   }
< }
16:06 DEBUG [databricks.sdk] Loaded from environment
16:06 DEBUG [databricks.sdk] Ignoring pat auth, because metadata-service is preferred
16:06 DEBUG [databricks.sdk] Ignoring basic auth, because metadata-service is preferred
16:06 DEBUG [databricks.sdk] Attempting to configure auth: metadata-service
16:06 INFO [databricks.sdk] Using Databricks Metadata Service authentication
16:06 DEBUG [databricks.labs.lsql.backends] [api][execute] CREATE TABLE IF NOT EXISTS hive_metastore.default.foo (first STRING NOT NULL, second BOOLEAN NOT... (18 more bytes)
16:06 DEBUG [databricks.labs.lsql.core] Executing SQL statement: CREATE TABLE IF NOT EXISTS hive_metastore.default.foo (first STRING NOT NULL, second BOOLEAN NOT NULL) USING DELTA
16:06 DEBUG [databricks.sdk] POST /api/2.0/sql/statements/
> {
>   "format": "JSON_ARRAY",
>   "statement": "CREATE TABLE IF NOT EXISTS hive_metastore.default.foo (first STRING NOT NULL, second BOOLEAN NOT... (18 more bytes)",
>   "warehouse_id": "TEST_DEFAULT_WAREHOUSE_ID"
> }
< 200 OK
< {
<   "statement_id": "01eef10b-016f-1931-9a1e-8029529a2db2",
<   "status": {
<     "error": {
<       "error_code": "BAD_REQUEST",
<       "message": "[INSUFFICIENT_PERMISSIONS] Insufficient privileges:\nUser does not have permission CREATE,USAGE o... (21 more bytes)"
<     },
<     "state": "FAILED"
<   }
< }
[gw8] linux -- Python 3.10.14 /home/runner/work/lsql/lsql/.venv/bin/python

Running from acceptance #59

@nfx
Copy link
Copy Markdown
Collaborator Author

nfx commented Apr 2, 2024

We can live with this single integration test failing, UCX bc-compat is caused by PySDK release, UCX is protected by a concrete dependency on SDK, and LSQL dependency on SDK is loose.

@nfx nfx merged commit 155bea0 into main Apr 2, 2024
@nfx nfx deleted the prepare/0.3.1 branch April 2, 2024 16:16
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

No reviews

Assignees

No one assigned

Labels

None yet

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

1 participant

@nfx