Skip to content

Databricks workflow with SQL Warehouse Pro crashes with http error 503 with version 1.7.4 #570

@OlofL

Description

@OlofL

Describe the bug

A workflow dbt job terminates with an error Max retries exceeded with url: ... (Caused by ResponseError('too many 503 error responses')) just when it starts sending SQL commands to the cluster.
This occurs only in connector version 1.7.4, not in version 1.7.3.
This occurs only with SQL Warehouse, not with SQL Warehouse Serverless

Steps To Reproduce

The problem occurs in a workflow in a databricks workspace with the following settings
Running on Azure, Databricks Premium, not Unity Catalog
Job cluster single node Standard_DS3_v2
Work cluster SQL Warehouse Pro X-Small, Cluster count: Active 0 Min 1 Max 1, Channel Current, Cost optimized
git source Azure Devops
Settings for library version dbt-databricks>=1.0.0,<2.0.0

Start the workflow. After the job cluster has been created and the SQL Warehouse has been started an error is shown in the log:

  • dbt build --profiles-dir ../misc/misc/ -t prod -f
    02:09:19 Running with dbt=1.7.5
    02:09:20 Registered adapter: databricks=1.7.4
    02:09:20 Unable to do partial parsing because saved manifest not found. Starting full parse.
    02:09:38 Found 393 models, 88 snapshots, 1 analysis, 5 seeds, 1443 tests, 109 sources, 8 exposures, 0 metrics, 920 macros, 0 groups, 0 semantic models
    02:09:38
    02:12:11
    02:12:11 Finished running in 0 hours 2 minutes and 32.34 seconds (152.34s).
    02:12:11 Encountered an error:
    Runtime Error
    HTTPSConnectionPool(host='[removed]', port=443): Max retries exceeded with url: /sql/1.0/warehouses/[removed] (Caused by ResponseError('too many 503 error responses'))

Changing to SQL Warehouse of type Serverless, X-Small, Cluster count: Active 0 Min 1 Max 1, Current solves the problem.
Running on older version of dbt-databricks library solves the problem (dbt-databricks>=1.0.0,<1.7.3)
The problem appeared on Jan 25 when the version was updated from 1.7.3 -> 1.7.4.
I have not been able to reproduce the same problem when starting dbt interactive, just as a workflow, but that can be my mistake, this has been a stressful couple days due to this.

Expected behavior

Log output:
02:08:27 Running with dbt=1.7.5
02:08:28 Registered adapter: databricks=1.7.3
02:08:28 Unable to do partial parsing because saved manifest not found. Starting full parse.
02:08:46 Found 393 models, 88 snapshots, 1 analysis, 5 seeds, 1443 tests, 109 sources, 8 exposures, 0 metrics, 919 macros, 0 groups, 0 semantic models
02:08:46
02:14:30 Concurrency: 12 threads (target='prod')
02:14:30
02:14:31 1 of 1835 START sql table model staging.rollup12helper ......................... [RUN]
02:14:31 2 of 1835 START sql table model staging.rollup24helper ......................... [RUN]

Screenshots and log output

N/A

System information

dbt 1.7.5
dbt-databricks 1.7.4

Configured in a dbt workflow, not much control over settings there.

Additional context

N/A

Metadata

Metadata

Assignees

No one assigned

    Labels

    bugSomething isn't working

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions