Skip to content

Remove unified flag usage, rely on host metadata#720

Open
hectorcast-db wants to merge 4 commits intomainfrom
hectorcast-db/stack/port-8-remove-unified-flag
Open

Remove unified flag usage, rely on host metadata#720
hectorcast-db wants to merge 4 commits intomainfrom
hectorcast-db/stack/port-8-remove-unified-flag

Conversation

@hectorcast-db
Copy link
Copy Markdown
Contributor

@hectorcast-db hectorcast-db commented Mar 19, 2026

🥞 Stacked PR

Use this link to review incremental changes.


Summary

Port of Go SDK #1547.

Removes HostType.UNIFIED and all runtime checks of experimentalIsUnifiedHost. Host type is now determined solely from URL pattern (accounts.* = ACCOUNTS, else WORKSPACE). Host metadata resolution (from earlier PRs in this stack) handles populating config fields automatically.

Key changes:

  • getHostType(): no longer returns UNIFIED; determined solely by URL pattern
  • isAccountClient(): no longer throws for unified hosts
  • getClientType(): simplified, no UNIFIED case
  • fetchDefaultOidcEndpoints(): removed unified OIDC branch, removed getUnifiedOidcEndpoints()
  • DatabricksCliCredentialsProvider.buildHostArgs(): removed --experimental-is-unified-host, --account-id, --workspace-id flags for unified case
  • AccountClient.getWorkspaceClient(): uses DNS zone matching (like Go SDK) to decide whether to reuse host or build deployment URL
  • HostType enum: removed UNIFIED value
  • authenticate(): removed X-Databricks-Org-Id header injection (handled by generated *Impl.java files)

Note: AccountClient.java is a generated file. The template needs to be updated.

NO_CHANGELOG=true

Test plan

  • All 1086 tests pass
  • UnifiedHostTest, DatabricksConfigTest, AccountClientTest, DatabricksCliCredentialsProviderTest updated

github-merge-queue bot pushed a commit that referenced this pull request Mar 19, 2026
## 🥞 Stacked PR

- [**#710 Add cloud field to
HostMetadata**](#710)
[[Files](https://github.com/databricks/databricks-sdk-java/pull/710/files)]
- [#711 Fix GetWorkspaceClient for unified account
hosts](#711)
[[Files](https://github.com/databricks/databricks-sdk-java/pull/711/files)]
- [#712 Add test for GetWorkspaceClient with SPOG
host](#712)
[[Files](https://github.com/databricks/databricks-sdk-java/pull/712/files)]
- [#713 Call resolveHostMetadata on Config
init](#713)
[[Files](https://github.com/databricks/databricks-sdk-java/pull/713/files)]
- [#714 Resolve TokenAudience from host metadata for account
hosts](#714)
[[Files](https://github.com/databricks/databricks-sdk-java/pull/714/files)]
- [#718 Make GCP SA token refresh
non-blocking](#718)
[[Files](https://github.com/databricks/databricks-sdk-java/pull/718/files)]
- [#719 Add integration test for host metadata
resolution](#719)
[[Files](https://github.com/databricks/databricks-sdk-java/pull/719/files)]
- [#720 Remove unified flag usage, rely on host
metadata](#720)
[[Files](https://github.com/databricks/databricks-sdk-java/pull/720/files)]

---------
## Summary

Port of Go SDK
[#1512](databricks/databricks-sdk-go#1512).

Adds a `cloud` field to `HostMetadata` that is populated from the
`/.well-known/databricks-config` discovery endpoint.

**Why:** Today, `isAws()`, `isAzure()`, and `isGcp()` infer cloud type
by suffix-matching the workspace hostname against a hardcoded list of
known DNS zones. This works for standard deployments but fails for
non-standard hostnames (custom vanity domains, unified hosts, etc.). The
discovery endpoint is the authoritative source and already returns a
`cloud` field, but the SDK was discarding it.

**Changes:**
- `HostMetadata`: new `cloud` field (`@JsonProperty("cloud")`), getter,
and 4-arg constructor
- `HostMetadataTest`: deserialization with/without cloud, constructor
tests

`NO_CHANGELOG=true`

## Test plan
- [x] `HostMetadataTest`: 4 tests for cloud field deserialization and
constructors
@hectorcast-db hectorcast-db force-pushed the hectorcast-db/stack/port-8-remove-unified-flag branch from 78816e5 to 483ae61 Compare March 19, 2026 11:53
@hectorcast-db hectorcast-db force-pushed the hectorcast-db/stack/port-7-integration-test-metadata branch from eebd798 to 4607961 Compare March 19, 2026 11:53
github-merge-queue bot pushed a commit that referenced this pull request Mar 19, 2026
## 🥞 Stacked PR

- [#710 Add cloud field to
HostMetadata](#710)
[[Files](https://github.com/databricks/databricks-sdk-java/pull/710/files)]
- [**#711 Fix GetWorkspaceClient for unified account
hosts**](#711)
[[Files](https://github.com/databricks/databricks-sdk-java/pull/711/files)]
- [#712 Add test for GetWorkspaceClient with SPOG
host](#712)
[[Files](https://github.com/databricks/databricks-sdk-java/pull/712/files)]
- [#713 Call resolveHostMetadata on Config
init](#713)
[[Files](https://github.com/databricks/databricks-sdk-java/pull/713/files)]
- [#714 Resolve TokenAudience from host metadata for account
hosts](#714)
[[Files](https://github.com/databricks/databricks-sdk-java/pull/714/files)]
- [#718 Make GCP SA token refresh
non-blocking](#718)
[[Files](https://github.com/databricks/databricks-sdk-java/pull/718/files)]
- [#719 Add integration test for host metadata
resolution](#719)
[[Files](https://github.com/databricks/databricks-sdk-java/pull/719/files)]
- [#720 Remove unified flag usage, rely on host
metadata](#720)
[[Files](https://github.com/databricks/databricks-sdk-java/pull/720/files)]

---------
## Summary

Port of Go SDK
[#1517](databricks/databricks-sdk-go#1517).

Fixes `getWorkspaceClient()` for unified account hosts that don't follow
the standard environment DNS zone pattern (e.g. SPOG/unified hosts).
Previously, the workspace host was always constructed via
`getDeploymentUrl(ws.getDeploymentName())`, which blindly appends the
environment's DNS zone. For unified hosts where the account and
workspace share the same host, this produces an incorrect URL.

**Changes:**
- `AccountClient.getWorkspaceClient()`: clones config instead of
mutating `this.config` for unified hosts

**Note:** `AccountClient.java` is a generated file. The template needs
to be updated.

`NO_CHANGELOG=true`

## Test plan
- [x] `AccountClientTest`: existing tests pass
@hectorcast-db hectorcast-db force-pushed the hectorcast-db/stack/port-8-remove-unified-flag branch from 483ae61 to ad7024c Compare March 23, 2026 09:28
@hectorcast-db hectorcast-db force-pushed the hectorcast-db/stack/port-7-integration-test-metadata branch from 4607961 to 153a890 Compare March 23, 2026 09:28
@hectorcast-db hectorcast-db force-pushed the hectorcast-db/stack/port-8-remove-unified-flag branch from ad7024c to b87fca5 Compare March 23, 2026 10:29
@hectorcast-db hectorcast-db force-pushed the hectorcast-db/stack/port-7-integration-test-metadata branch from 153a890 to 821337f Compare March 23, 2026 10:29
@hectorcast-db hectorcast-db force-pushed the hectorcast-db/stack/port-7-integration-test-metadata branch from 821337f to f79a3e8 Compare March 23, 2026 13:04
@hectorcast-db hectorcast-db force-pushed the hectorcast-db/stack/port-8-remove-unified-flag branch from 72fae6f to 34394bc Compare March 30, 2026 12:57
github-merge-queue bot pushed a commit that referenced this pull request Mar 31, 2026
## 🥞 Stacked PR
Use this
[link](https://github.com/databricks/databricks-sdk-java/pull/719/files)
to review incremental changes.
-
[**hectorcast-db/stack/port-7-integration-test-metadata**](#719)
[[Files
changed](https://github.com/databricks/databricks-sdk-java/pull/719/files)]
-
[hectorcast-db/stack/port-8-remove-unified-flag](#720)
[[Files
changed](https://github.com/databricks/databricks-sdk-java/pull/720/files/cbd81a9a8534a735250b64fe3957a3923b6da899..34394bc5a5eb92745574c42c336a326c8719cebd)]

---------
## Summary

Port of Go SDK
[#1546](databricks/databricks-sdk-go#1546).

Adds an integration test verifying that `config.resolve()` populates
`accountId`, `workspaceId`, and `discoveryUrl` from the host's
`/.well-known/databricks-config` endpoint.

**Changes:**
- `HostMetadataIT`: integration test for metadata resolution (requires
`DATABRICKS_HOST` via `ucws` env context)

`NO_CHANGELOG=true`

## Test plan
- [x] All unit tests pass (no production code changes)
- [x] `HostMetadataIT` runs against live workspace
Base automatically changed from hectorcast-db/stack/port-7-integration-test-metadata to main March 31, 2026 06:50
Port of Go SDK #1547. Removes HostType.UNIFIED and all runtime checks
of experimentalIsUnifiedHost. Host type is now determined solely from
URL pattern (accounts.* = ACCOUNTS, else WORKSPACE). Host metadata
resolution from /.well-known/databricks-config (added in PR 4) handles
populating accountId, workspaceId, and discoveryUrl automatically.

Key changes:
- getHostType(): no longer returns UNIFIED
- isAccountClient(): no longer throws for unified hosts
- getClientType(): simplified, no UNIFIED case
- fetchDefaultOidcEndpoints(): removed unified OIDC branch
- DatabricksCliCredentialsProvider: removed --experimental-is-unified-host
- AccountClient.getWorkspaceClient(): uses DNS zone matching (like Go SDK)
  to decide whether to reuse host or build deployment URL

Co-authored-by: Isaac
@hectorcast-db hectorcast-db force-pushed the hectorcast-db/stack/port-8-remove-unified-flag branch from 34394bc to 849be46 Compare March 31, 2026 06:52
Remove the experimentalIsUnifiedHost gate in tryResolveHostMetadata()
so metadata is resolved unconditionally during config init. Add
auto-stub for /.well-known/databricks-config in FixtureServer to
prevent metadata resolution from interfering with unrelated tests.
Update tests to remove redundant explicit resolveHostMetadata() calls
and unnecessary setExperimentalIsUnifiedHost(true).

Co-authored-by: Isaac
After removing the unified-host guard, all configs now resolve
/.well-known/databricks-config. Update HttpPathTest and
IdempotencyTestingAPITest mocks to account for this extra call.

Co-authored-by: Isaac
@hectorcast-db
Copy link
Copy Markdown
Contributor Author

Range-diff: main (884e9dd -> 27eafd3)
databricks-sdk-java/src/test/java/com/databricks/sdk/service/gentesting/unittests/HttpPathTest.java
@@ -0,0 +1,13 @@
+diff --git a/databricks-sdk-java/src/test/java/com/databricks/sdk/service/gentesting/unittests/HttpPathTest.java b/databricks-sdk-java/src/test/java/com/databricks/sdk/service/gentesting/unittests/HttpPathTest.java
+--- a/databricks-sdk-java/src/test/java/com/databricks/sdk/service/gentesting/unittests/HttpPathTest.java
++++ b/databricks-sdk-java/src/test/java/com/databricks/sdk/service/gentesting/unittests/HttpPathTest.java
+         .thenAnswer(
+             invocation -> {
+               Request request = invocation.getArgument(0);
++              // Handle host metadata discovery call
++              if (request.getUri().toString().equals(HOST + "/.well-known/databricks-config")) {
++                return new Response("{}", 200, "OK", new URL(HOST));
++              }
+               String expectedUrl = HOST + testCase.path;
+               if (!request.getUri().toString().equals(expectedUrl)) {
+                 throw new AssertionError(
\ No newline at end of file
databricks-sdk-java/src/test/java/com/databricks/sdk/service/gentesting/unittests/IdempotencyTestingAPITest.java
@@ -0,0 +1,31 @@
+diff --git a/databricks-sdk-java/src/test/java/com/databricks/sdk/service/gentesting/unittests/IdempotencyTestingAPITest.java b/databricks-sdk-java/src/test/java/com/databricks/sdk/service/gentesting/unittests/IdempotencyTestingAPITest.java
+--- a/databricks-sdk-java/src/test/java/com/databricks/sdk/service/gentesting/unittests/IdempotencyTestingAPITest.java
++++ b/databricks-sdk-java/src/test/java/com/databricks/sdk/service/gentesting/unittests/IdempotencyTestingAPITest.java
+   void testIdempotencyAutoRequestID(AutoRequestIDTestCase testCase) throws Exception {
+     // Setup: Create DummyHttpClient with sequential responses for retries
+     DummyHttpClient realClient = new DummyHttpClient();
++    // Add a response for the host metadata discovery call
++    Request metadataRequest =
++        new Request("GET", "https://test.databricks.com/.well-known/databricks-config");
++    realClient.with(
++        metadataRequest,
++        new Response(metadataRequest, 200, "OK", Collections.emptyMap(), "{}"));
+     for (Response response : testCase.httpResponses) {
+       realClient.with(testCase.httpRequest, response);
+     }
+       assertEquals(testCase.wantResult, result, "Test case: " + testCase.name);
+ 
+       // Capture and verify request IDs
++      // 3 calls: host metadata discovery + 2 API calls (initial + retry)
+       ArgumentCaptor<Request> requestCaptor = ArgumentCaptor.forClass(Request.class);
+-      verify(spyClient, times(2)).execute(requestCaptor.capture());
++      verify(spyClient, times(3)).execute(requestCaptor.capture());
+ 
+       List<Request> capturedRequests = requestCaptor.getAllValues();
+-      String firstRequestId = capturedRequests.get(0).getQuery().get("request_id").get(0);
+-      String secondRequestId = capturedRequests.get(1).getQuery().get("request_id").get(0);
++      String firstRequestId = capturedRequests.get(1).getQuery().get("request_id").get(0);
++      String secondRequestId = capturedRequests.get(2).getQuery().get("request_id").get(0);
+ 
+       assertNotNull(firstRequestId, "Auto-generated request_id should not be null");
+       assertFalse(firstRequestId.isEmpty(), "Auto-generated request_id should not be empty");
\ No newline at end of file

Reproduce locally: git range-diff f28430b..884e9dd f28430b..27eafd3 | Disable: git config gitstack.push-range-diff false

@hectorcast-db
Copy link
Copy Markdown
Contributor Author

Range-diff: main (27eafd3 -> 7810145)
databricks-sdk-java/src/test/java/com/databricks/sdk/core/UnifiedHostTest.java
@@ -96,8 +96,9 @@
    }
  
 -  // --- OIDC Endpoint Tests ---
--
--  @Test
++  // --- isAccountClient() Tests ---
+ 
+   @Test
 -  public void testOidcEndpointsForUnifiedHost() throws IOException {
 -    DatabricksConfig config =
 -        new DatabricksConfig()
@@ -114,9 +115,8 @@
 -        "https://unified.databricks.com/oidc/accounts/test-account-123/v1/token",
 -        endpoints.getTokenEndpoint());
 -  }
-+  // --- isAccountClient() Tests ---
- 
-   @Test
+-
+-  @Test
 -  public void testOidcEndpointsForUnifiedHostMissingAccountId() {
 -    DatabricksConfig config =
 -        new DatabricksConfig()
@@ -159,9 +159,7 @@
 +    // Non-accounts hosts are not account clients
      assertFalse(
 -        new DatabricksConfig().setHost("https://adb-123.azuredatabricks.net").isAccountClient());
-+        new DatabricksConfig()
-+            .setHost("https://mycompany.databricks.com")
-+            .isAccountClient());
++        new DatabricksConfig().setHost("https://mycompany.databricks.com").isAccountClient());
    }
  
    // --- Environment Variable Tests ---
databricks-sdk-java/src/test/java/com/databricks/sdk/core/oauth/ExternalBrowserCredentialsProviderTest.java
@@ -0,0 +1,31 @@
+diff --git a/databricks-sdk-java/src/test/java/com/databricks/sdk/core/oauth/ExternalBrowserCredentialsProviderTest.java b/databricks-sdk-java/src/test/java/com/databricks/sdk/core/oauth/ExternalBrowserCredentialsProviderTest.java
+--- a/databricks-sdk-java/src/test/java/com/databricks/sdk/core/oauth/ExternalBrowserCredentialsProviderTest.java
++++ b/databricks-sdk-java/src/test/java/com/databricks/sdk/core/oauth/ExternalBrowserCredentialsProviderTest.java
+ 
+   @Test
+   void openIDConnectEndPointsTestAccounts() throws IOException {
++    HttpClient mockHttpClient = Mockito.mock(HttpClient.class);
++    // Mock the host metadata call (resolve) to return 404, matching FixtureServer auto-stub
++    // behavior. The actual OIDC endpoints for account clients are computed locally without HTTP.
++    Mockito.doAnswer(
++            invocation -> {
++              Request req = invocation.getArgument(0);
++              if (req.getUrl().contains("/.well-known/databricks-config")) {
++                return new Response(
++                    "{\"error_code\":\"NOT_FOUND\",\"message\":\"not found\"}",
++                    new URL(req.getUrl()));
++              }
++              throw new IOException("Unexpected request: " + req.getUrl());
++            })
++        .when(mockHttpClient)
++        .execute(any(Request.class));
++
+     DatabricksConfig config =
+         new DatabricksConfig()
+             .setAuthType("external-browser")
+             .setHost("https://accounts.cloud.databricks.com")
+-            .setHttpClient(new CommonsHttpClient.Builder().withTimeoutSeconds(30).build())
++            .setHttpClient(mockHttpClient)
+             .setAccountId("testAccountId");
+     config.resolve();
+ 
\ No newline at end of file
databricks-sdk-java/src/test/java/com/databricks/sdk/service/gentesting/unittests/IdempotencyTestingAPITest.java
@@ -8,8 +8,7 @@
 +    Request metadataRequest =
 +        new Request("GET", "https://test.databricks.com/.well-known/databricks-config");
 +    realClient.with(
-+        metadataRequest,
-+        new Response(metadataRequest, 200, "OK", Collections.emptyMap(), "{}"));
++        metadataRequest, new Response(metadataRequest, 200, "OK", Collections.emptyMap(), "{}"));
      for (Response response : testCase.httpResponses) {
        realClient.with(testCase.httpRequest, response);
      }

Reproduce locally: git range-diff f28430b..27eafd3 f28430b..7810145 | Disable: git config gitstack.push-range-diff false

@hectorcast-db hectorcast-db marked this pull request as ready for review March 31, 2026 10:12
The openIDConnectEndPointsTestAccounts test was making real HTTP calls
to accounts.cloud.databricks.com, causing failures when the endpoint
returned HTML instead of JSON. Use a mock HttpClient that stubs the
host metadata call, since account OIDC endpoints are computed locally.
Also apply spotless formatting fixes.

Co-authored-by: Isaac
@hectorcast-db hectorcast-db force-pushed the hectorcast-db/stack/port-8-remove-unified-flag branch from 7810145 to 19aa378 Compare March 31, 2026 10:14
@hectorcast-db
Copy link
Copy Markdown
Contributor Author

Range-diff: main (7810145 -> 19aa378)
databricks-sdk-java/src/test/java/com/databricks/sdk/core/DatabricksConfigTest.java
@@ -132,9 +132,11 @@
        DatabricksConfig config = new DatabricksConfig().setHost(server.getUrl());
        config.resolve(emptyEnv());
        DatabricksException ex =
+     }
+   }
  
-   @Test
-   public void testResolveHostMetadataSetsTokenAudienceForAccountHost() throws IOException {
+-  @Test
+-  public void testResolveHostMetadataSetsTokenAudienceForAccountHost() throws IOException {
 -    // For a unified host with no workspaceId (ACCOUNT client type), resolveHostMetadata should
 -    // set tokenAudience to accountId when not already configured.
 -    String response =
@@ -157,19 +159,11 @@
 -      config.resolveHostMetadata();
 -      assertEquals(DUMMY_ACCOUNT_ID, config.getTokenAudience());
 -    }
-+    // For an account host, resolveHostMetadata should set tokenAudience to accountId
-+    // when not already configured. We verify the preconditions here.
-+    DatabricksConfig accountConfig =
-+        new DatabricksConfig()
-+            .setHost("https://accounts.cloud.databricks.com")
-+            .setAccountId(DUMMY_ACCOUNT_ID);
-+    assertEquals(ClientType.ACCOUNT, accountConfig.getClientType());
-+    assertNull(accountConfig.getTokenAudience());
-+    // When resolve runs with a reachable host, tryResolveHostMetadata will call
-+    // resolveHostMetadata which sets tokenAudience = accountId for ACCOUNT clients.
-   }
- 
+-  }
+-
    @Test
+   public void testResolveHostMetadataDoesNotOverwriteTokenAudience() throws IOException {
+     String response =
                .setAccountId(DUMMY_ACCOUNT_ID)
                .setTokenAudience("custom-audience");
        config.resolve(emptyEnv());

Reproduce locally: git range-diff f28430b..7810145 f28430b..19aa378 | Disable: git config gitstack.push-range-diff false

@github-actions
Copy link
Copy Markdown

If integration tests don't run automatically, an authorized user can run them manually by following the instructions below:

Trigger:
go/deco-tests-run/sdk-java

Inputs:

  • PR number: 720
  • Commit SHA: 19aa37833544f7d7f6dba7fafeb83bda12fbc6c8

Checks will be approved automatically on success.

Copy link
Copy Markdown
Contributor

@tejaskochar-db tejaskochar-db left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM mod comment.

Comment on lines 40 to 45
/**
* Flag to explicitly mark a host as a unified host. Note: This API is experimental and may change
* or be removed in future releases without notice.
*/
@ConfigAttribute(env = "DATABRICKS_EXPERIMENTAL_IS_UNIFIED_HOST")
private Boolean experimentalIsUnifiedHost;
Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I'm guessing these can't be removed as it will break users that had set it? Maybe should still document that it is no longer used here?

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants