Skip to content

Commit dea1802

Browse files
djouallahclaude
andcommitted
Workload: LakeSail joins as the fifth leg, 10 files a run
dbt-spark method: session on Sail main acfa37959f, its own tree (models/aemo/sail, tests/aemo/sail) copied from gluten with Sail's SQL written in: GROUP BY ordinals (lakehq/sail#2836) and the interval joined on epoch seconds (apache/datafusion#26003). The gluten tree is back to its pre-probe text; the shared Spark macros keep their target.name == 'sail' branches. The probe workflow is gone. First green: probe run 39692966782, 6/6 models, 39/39 tests. Co-Authored-By: Claude Opus 5.5 <[email protected]>
1 parent 579e9a2 commit dea1802

36 files changed

Lines changed: 581 additions & 146 deletions

‎.github/scripts/workload_run_engine.py‎

Lines changed: 4 additions & 5 deletions
Original file line numberDiff line numberDiff line change
@@ -1,7 +1,7 @@
11
"""Run the dbt workload on ONE engine and write its part: `python workload_run_engine.py parts`.
22
3-
The engine is BENCH_ENGINE, a target in workload/profiles.yml: duckdb, polars, gluten or trino --
4-
or `sail`, the probe target, run only by workload_sail_probe.yml.
3+
The engine is BENCH_ENGINE, a target in workload/profiles.yml: duckdb, polars, gluten, trino or
4+
sail.
55
66
1. mint the OneLake bearer and hand it to the profile through the environment
77
2. gluten only: start bench's Gluten/Velox SparkSession (the ETL's writing variant, so the
@@ -44,7 +44,6 @@
4444
BENCH_ENGINE,
4545
DEFAULT_PROCESS_LIMIT,
4646
ENGINES,
47-
PROBE_ENGINES,
4847
WorkloadConfig,
4948
)
5049

@@ -324,8 +323,8 @@ def process_limit(engine: str) -> int:
324323
def main(parts: Path) -> int:
325324
cfg = WorkloadConfig.from_env()
326325
engine = cfg.engine
327-
if engine not in ENGINES + PROBE_ENGINES:
328-
raise SystemExit(f"BENCH_ENGINE must be one of {ENGINES + PROBE_ENGINES}, not {engine!r}")
326+
if engine not in ENGINES:
327+
raise SystemExit(f"BENCH_ENGINE must be one of {ENGINES}, not {engine!r}")
329328
parts.mkdir(parents=True, exist_ok=True)
330329
started = datetime.now(UTC).strftime("%Y-%m-%dT%H:%M:%SZ")
331330
part = {

‎.github/workflows/workload.yml‎

Lines changed: 17 additions & 8 deletions
Original file line numberDiff line numberDiff line change
@@ -3,7 +3,7 @@ name: dbt workload
33
# THE "MY WORKLOAD" TEST: the fabric-dbt-examples dbt project (workload/, see workload/SOURCE),
44
# run by bench on its own engines and auth, Iceberg only.
55
#
6-
# land -> dbt-Iceberg (matrix: duckdb, polars, gluten, trino; one at a time) -> grade
6+
# land -> dbt-Iceberg (matrix: duckdb, polars, gluten, trino, sail; one at a time) -> grade
77
#
88
# ON A SCHEDULE, EVERY 2 HOURS, NEVER RESET: new AEMO files land every run and each engine folds
99
# process_limit of them into its own tables, which only grow. The Real workload tab shows how far
@@ -19,15 +19,15 @@ on:
1919
workflow_dispatch:
2020
inputs:
2121
engines:
22-
description: Comma-separated dbt targets (duckdb, polars, gluten, trino).
22+
description: Comma-separated dbt targets (duckdb, polars, gluten, trino, sail).
2323
type: string
24-
default: duckdb,polars,gluten,trino
24+
default: duckdb,polars,gluten,trino,sail
2525
process_limit:
2626
description: Files each fact model folds per run, newest first. A backlog converges over runs.
2727
type: string
2828
default: "50"
2929
process_limits:
30-
description: Per-engine files a run, e.g. "gluten=100,trino=150"; empty uses each engine's proven batch (gluten=360,duckdb=360,trino=100,polars=50). Engines not named use process_limit.
30+
description: Per-engine files a run, e.g. "gluten=100,trino=150"; empty uses each engine's proven batch (gluten=360,duckdb=360,trino=100,polars=50,sail=10). Engines not named use process_limit.
3131
type: string
3232
default: ""
3333
daily_download_limit:
@@ -82,7 +82,7 @@ jobs:
8282

8383
- id: plan
8484
run: |
85-
ENGINES=$(printf '%s' "${{ inputs.engines || 'duckdb,polars,gluten,trino' }}" \
85+
ENGINES=$(printf '%s' "${{ inputs.engines || 'duckdb,polars,gluten,trino,sail' }}" \
8686
| jq -Rc 'split(",") | map(sub("^\\s+";"") | sub("\\s+$";"")) | map(select(length > 0))')
8787
echo "engines=$ENGINES" >> "$GITHUB_OUTPUT"
8888
echo "started=$(date -u +%Y-%m-%dT%H:%M:%SZ)" >> "$GITHUB_OUTPUT"
@@ -125,6 +125,7 @@ jobs:
125125
|| matrix.engine == 'polars' && '🐻‍❄️ dbt-polars'
126126
|| matrix.engine == 'gluten' && '🚀 dbt-spark on Gluten/Velox'
127127
|| matrix.engine == 'trino' && '🐰 dbt-trino'
128+
|| matrix.engine == 'sail' && '⛵ dbt-spark on LakeSail'
128129
|| matrix.engine }}
129130
runs-on: ubuntu-latest
130131
# The OneLake bearer is baked into the profile (DuckDB's ATTACH, pyiceberg's catalog) and
@@ -144,8 +145,9 @@ jobs:
144145
process_limit: ${{ inputs.process_limit || '50' }}
145146
# Each engine's largest batch that holds on the schedule as its tables grow (2026-10-21):
146147
# Spark 360 (39403578519), DuckDB 360 with the TEMP VIEW batch (39427985965), Trino 100 (its
147-
# 9 GB per-query cap failed 360), Polars 50: at 75 the runner died (39493139124).
148-
PROCESS_LIMITS: ${{ inputs.process_limits || 'gluten=360,duckdb=360,trino=100,polars=50' }}
148+
# 9 GB per-query cap failed 360), Polars 50: at 75 the runner died (39493139124). Sail 10, its
149+
# only proven batch so far (probe run 39692966782: 5.6 GB peak RSS, 9.1 GB spilled).
150+
PROCESS_LIMITS: ${{ inputs.process_limits || 'gluten=360,duckdb=360,trino=100,polars=50,sail=10' }}
149151
AZURE_TRANSPORT_OPTION_TYPE: curl
150152
CURL_CA_INFO: /etc/ssl/certs/ca-certificates.crt
151153
DUCKDB_TEMP_DIR: /mnt/duckdb_spill
@@ -168,7 +170,7 @@ jobs:
168170
- name: Free disk
169171
run: |
170172
sudo rm -rf /usr/share/dotnet /usr/local/lib/android /opt/ghc /opt/hostedtoolcache/CodeQL /usr/local/share/boost
171-
sudo mkdir -p /mnt/duckdb_spill /mnt/polars_spill /mnt/spark_spill && sudo chown "$USER" /mnt/duckdb_spill /mnt/polars_spill /mnt/spark_spill
173+
sudo mkdir -p /mnt/duckdb_spill /mnt/polars_spill /mnt/spark_spill /mnt/sail_spill && sudo chown "$USER" /mnt/duckdb_spill /mnt/polars_spill /mnt/spark_spill /mnt/sail_spill
172174
df -h / /mnt
173175
- uses: actions/checkout@v5
174176
- uses: actions/setup-python@v6
@@ -206,6 +208,13 @@ jobs:
206208
~/.m2/repository
207209
key: ivy-spark41-iceberg111-v1
208210

211+
# Sail spills to the OS temp dir, and /tmp has a quota: fct_scada's merge failed "Disk quota
212+
# exceeded (os error 122)" (run 39691469943). Sail's leg only: an empty TMPDIR elsewhere would
213+
# not mean /tmp to every engine.
214+
- name: Sail's spill on /mnt
215+
if: matrix.engine == 'sail'
216+
run: echo "TMPDIR=/mnt/sail_spill" >> "$GITHUB_ENV"
217+
209218
# The official image bench/trino.py pins, pulled before the build.
210219
- name: Pull the Trino image
211220
if: matrix.engine == 'trino'

‎.github/workflows/workload_sail_probe.yml‎

Lines changed: 0 additions & 83 deletions
This file was deleted.

‎LEARNING.md‎

Lines changed: 4 additions & 4 deletions
Original file line numberDiff line numberDiff line change
@@ -810,11 +810,11 @@ so they share one history. First all-green run:
810810
- **dbt-polars tests need `azure/login`.** They read through pyiceberg's ADLS FileIO with no
811811
credential, which falls back to azure-identity's default chain.
812812

813-
### dbt on Sail (probe)
813+
### dbt on Sail
814814

815-
`workload_sail_probe.yml` runs the gluten tree through dbt-spark `method: session` on Sail's
816-
in-process Spark Connect server (pysail main 151a417d5d), into `dbt_sail_*`. It is not a leg:
817-
workload.yml, `ENGINES` and the page do not know it. First green run, 10 files a model:
815+
The fifth leg (2026-10-22), first run as a probe: dbt-spark `method: session` on Sail's in-process
816+
Spark Connect server (pysail main, a tmp_sail_main.yml build), its own tree copied from the gluten
817+
one, into `dbt_sail_*`, 10 files a run. First green run (probe, main 151a417d5d), 10 files a model:
818818
[39692966782](https://github.com/djouallah/lakehouse_benchmark/actions/runs/39692966782),
819819
6/6 models and 39/39 tests (6 warns, the same ones the legs have), every merge incremental,
820820
283 s, 5.6 GB peak RSS, 9.1 GB spilled. What it took, each on the `sail` target only:

‎bench/workload/config.py‎

Lines changed: 2 additions & 6 deletions
Original file line numberDiff line numberDiff line change
@@ -14,7 +14,7 @@
1414

1515
# The dbt targets, in run order. Same names as workload/profiles.yml and the engine folders under
1616
# workload/models/aemo/.
17-
ENGINES = ("duckdb", "polars", "gluten", "trino")
17+
ENGINES = ("duckdb", "polars", "gluten", "trino", "sail")
1818

1919
# bench's engine id for each target, so labels and colours match the rest of the page once it is
2020
# shown there.
@@ -26,10 +26,6 @@
2626
"sail": "lakesail_iceberg",
2727
}
2828

29-
# Targets workload_run_engine.py also runs, but only from their own probe workflow
30-
# (workload_sail_probe.yml): never in workload.yml's matrix, never published.
31-
PROBE_ENGINES = ("sail",)
32-
3329

3430
# The landing zone, inside bench's lakehouse. Its own folder: the ETL's Files/csv holds only the
3531
# daily CSVs, flat, while this project reads csv_raw/{daily,scada_today,price_today,duid} plus the
@@ -53,7 +49,7 @@ def files_abfss(self) -> str:
5349

5450
@property
5551
def estimated_gib(self) -> float:
56-
"""What an engine sizes its memory against (bench's Sail engine, the sail probe: sort-merge
52+
"""What an engine sizes its memory against (bench's Sail engine, the sail leg: sort-merge
5753
joins past half its 10 GiB pool). Not an SF: the workload's tables only grow -- fct_scada is
5854
past 380M rows -- so this says "bigger than memory". With hash joins the runner died in
5955
fct_scada's merge (run 39690698607): a hash join's build side cannot spill."""

‎docs/index.html‎

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -1114,7 +1114,7 @@ <h3>Tried, not added</h3>
11141114

11151115
// The Real workload tab: engines by maturity, not speed. The Spark leg runs on Gluten/Velox, so it is named
11161116
// that, not "Spark", which would read as vanilla Spark.
1117-
const WORKLOAD_ORDER = [GLUTEN, "trino_iceberg", "duckdb_iceberg", "polars_iceberg"];
1117+
const WORKLOAD_ORDER = [GLUTEN, "trino_iceberg", "duckdb_iceberg", "polars_iceberg", "lakesail_iceberg"];
11181118
const wlName = e => LABEL[e] ?? e;
11191119

11201120
function renderRealWorkload() {

‎requirements/workload_sail.txt‎

Lines changed: 4 additions & 5 deletions
Original file line numberDiff line numberDiff line change
@@ -1,12 +1,11 @@
1-
# The sail PROBE (workload_sail_probe.yml), not a leg of workload.yml: dbt-spark `method: session`
2-
# on Sail's in-process Spark Connect server, running the gluten model tree.
1+
# The sail leg: dbt-spark `method: session` on Sail's in-process Spark Connect server.
32
#
43
# dbt-spark WITHOUT the [session] extra: it would pull pyspark, and Sail needs pyspark-client and
54
# never pyspark (requirements/lakesail_iceberg.txt). Session mode imports pyspark at run time only.
65
dbt-spark
76
azure-identity>=1.19
87
psutil>=6
9-
# Sail main (lakehq/sail@151a417d5d, tmp_sail_main.yml), as the ETL runs it: the stage table's
10-
# `file` comes from input_file_name() (lakehq/sail#2571), which 0.7.2 lacks.
11-
pysail @ https://github.com/djouallah/lakehouse_benchmark/releases/download/sail-main-151a417d5d/pysail-0.7.2%2Bmain151a417d5d-cp38-abi3-manylinux_2_17_x86_64.manylinux2014_x86_64.whl
8+
# Sail main (lakehq/sail@acfa37959f, tmp_sail_main.yml), until a release after 0.7.2: the stage
9+
# table's `file` comes from input_file_name() (lakehq/sail#2571).
10+
pysail @ https://github.com/djouallah/lakehouse_benchmark/releases/download/sail-main-acfa37959f/pysail-0.7.2%2Bmainacfa37959f-cp38-abi3-manylinux_2_17_x86_64.manylinux2014_x86_64.whl
1211
pyspark-client

‎workload/dbt_project.yml‎

Lines changed: 9 additions & 3 deletions
Original file line numberDiff line numberDiff line change
@@ -5,6 +5,7 @@
55
# polars dbt-polars, pyiceberg REST catalog (+ an Azure storage catalog over the landing)
66
# gluten dbt-spark `method: session` on Spark + Gluten/Velox, Iceberg (port of the Delta `spark`)
77
# trino dbt-trino on bench's Trino container, Iceberg (ported from the gluten tree)
8+
# sail dbt-spark `method: session` on LakeSail over Spark Connect, Iceberg (from the gluten tree)
89
#
910
# ONE PROJECT, ONE TREE PER ENGINE: models/aemo/<engine>/<layer>/<model>.sql. Exactly one tree
1011
# is +enabled per run, keyed on target.name, so the same model names never collide. There is
@@ -60,9 +61,13 @@ models:
6061
+enabled: "{{ target.name == 'polars' }}"
6162
dimensions: {+schema: mart}
6263
marts: {+schema: landing}
63-
# `sail` is the probe target (workload_sail_probe.yml): the same Spark SQL tree on Sail.
6464
gluten:
65-
+enabled: "{{ target.name == 'gluten' or target.name == 'sail' }}"
65+
+enabled: "{{ target.name == 'gluten' }}"
66+
+file_format: iceberg
67+
dimensions: {+schema: mart}
68+
marts: {+schema: landing}
69+
sail:
70+
+enabled: "{{ target.name == 'sail' }}"
6671
+file_format: iceberg
6772
dimensions: {+schema: mart}
6873
marts: {+schema: landing}
@@ -84,5 +89,6 @@ data_tests:
8489
# singular tests under tests/aemo/<engine>/ do have one, and are gated on it.
8590
duckdb: {+enabled: "{{ target.name == 'duckdb' }}"}
8691
polars: {+enabled: "{{ target.name == 'polars' }}"}
87-
gluten: {+enabled: "{{ target.name == 'gluten' or target.name == 'sail' }}"}
92+
gluten: {+enabled: "{{ target.name == 'gluten' }}"}
93+
sail: {+enabled: "{{ target.name == 'sail' }}"}
8894
trino: {+enabled: "{{ target.name == 'trino' }}"}

‎workload/macros/gluten_create_table_as.sql‎

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -29,7 +29,7 @@
2929
{%- set view = relation.identifier ~ '__dbt_ctas' -%}
3030
{%- do run_query('create or replace temporary view ' ~ view ~ ' as\n' ~ compiled_code) -%}
3131
{%- set columns = [] -%}
32-
{#-- `describe table`: Sail's parser wants the keyword Spark makes optional (the sail probe, run
32+
{#-- `describe table`: Sail's parser wants the keyword Spark makes optional (the sail leg, run
3333
39685977867: "found dim_duid__dbt_ctas ... expected ... 'TABLE'"). --#}
3434
{%- for row in run_query(('describe table ' if target.name == 'sail' else 'describe ') ~ view) -%}
3535
{%- if row[0] and not row[0].startswith('#') -%}

‎workload/macros/merge_prune.sql‎

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -21,7 +21,7 @@
2121
{%- set keys = [key] if key is string else (key or []) -%}
2222
{%- set col = (['file', 'date'] | select('in', keys) | list | first) -%}
2323
{%- if not col or not execute -%}{{ return([]) }}{%- endif -%}
24-
{#-- The sail probe target casts the collected column: Sail returned `file` as Arrow string_view
24+
{#-- The sail leg casts the collected column: Sail returned `file` as Arrow string_view
2525
here, which the Spark Connect client cannot read ("[UNSUPPORTED_DATA_TYPE_FOR_ARROW_CONVERSION]
2626
string_view is not supported in conversion to Arrow", runs 39687083463 and 39687830566, with
2727
pyspark-client 4.2.0 and 4.1.1): DISTINCT over substring_index, lakehq/sail#2835. --#}

0 commit comments

Comments
 (0)