Skip to content

Instantly share code, notes, and snippets.

@yihua
Last active November 3, 2026 08:48
Show Gist options
  • Select an option

  • Save yihua/66ad80224f8e50020025dc76d138dd0a to your computer and use it in GitHub Desktop.

Select an option

Save yihua/66ad80224f8e50020025dc76d138dd0a to your computer and use it in GitHub Desktop.
Reproducing the hudi-rs upsert benchmark against delta-rs and PyIceberg

Reproducing the hudi-rs upsert benchmark against delta-rs and PyIceberg

This guide reruns the single-node benchmark behind the numbers in the Apache Hudi Python/Rust quick start. It compares hudi-rs, delta-rs and PyIceberg on the same workloads, through each library's Python API, and checks every end state for correctness.

Everything is pinned:

  • Harness: benchmark/upsert at hudi-rs commit 543642771a1440f8470053831463052e190b3923. It generates the data, runs every case in its own processes, gates on machine load, checks correctness and records timings.
  • Libraries: requirements-lock.txt in the code gist: hudi 0.6.0, deltalake 1.6.6, pyiceberg 0.12.0, pyarrow 26.0.0, and every transitive dependency.
  • delta-rs and PyIceberg cases: compare_cases.py in the same gist (revision d94a68449a1d), loaded into the harness with --extra-cases.

What is compared

Case Library Table
hudi-mor hudi-rs merge-on-read, record-level index on
hudi-cow hudi-rs copy-on-write, record-level index on (the HudiTable.create defaults)
hudi-cow-noindex hudi-rs copy-on-write, record-level index off
delta deltalake defaults
iceberg pyiceberg defaults, SQL catalog over sqlite

Each library keeps its own defaults for file size and compression: hudi-rs 120 MiB files and zstd, delta-rs 100 MiB files and snappy, PyIceberg 512 MiB of in-memory Arrow per file and zstd. The only options set are the ones a workload needs: the record key and ordering field for hudi-rs and the identifier fields for PyIceberg's upsert. delta-rs's change data feed stays off, its default. No table is partitioned.

Each operation is the call the library documents for it:

hudi-rs delta-rs PyIceberg
Load HudiTable.create(...), then append(table) write_deltalake(path, table) create_table(...), then append(table)
Upsert upsert(table) merge(source, "t.k = s.k").when_matched_update_all().when_not_matched_insert_all().execute() upsert(table, join_cols=[k])
Delete by key delete_keys(keys) merge(keys, "t.k = s.k").when_matched_delete().execute() delete(delete_filter=In(k, keys))
Read read(), data columns only to_pyarrow_table() scan().to_arrow()

References: delta-rs merging tables and deleting rows, PyIceberg API. delta-rs documents two ways to delete by key; the merge form was faster on these workloads (2.9 s against 4.2 s per uuid delete commit), so it is the one used. Every read returns a fully materialized Arrow table with the same data columns; Hudi's five _hoodie_* metadata columns are not read.

Workloads

Each scenario generates its base table and every commit's input from a fixed seed before anything is timed, so every library receives the same rows. A run loads the base table, applies 5 upsert commits and 5 delete commits, then reads the whole table back.

Scenario Models Base rows Each commit
rag-chunks Re-syncing a retrieval corpus (text + 384-float embedding) 1M upsert 10,000 chunks, delete 1,000
training-set Curating a fine-tuning set 1M upsert 10,000 examples, delete 1,000
uuid Random-key upserts 10M upsert 10,000 existing + 1,000 new keys, delete 1,000
nyc-taxi Correcting a public dataset (NYC yellow taxi, January 2022) 2.5M upsert 1% of trips, delete 0.1%
tpcds-web-returns A warehouse fact table, composite key 0.7M upsert 1% matched + 10% new, delete 5%

training-set also times an incremental read and a time-travel read after every commit. Those reads are not in the published table and are not like-for-like: hudi-rs and PyIceberg read the commit's inserted and updated rows, while delta-rs, with its change data feed off, compares the two snapshots. Turning the feed on did not change delta-rs's load, commit or read times on this workload.

Keys are random, so every copy-on-write commit touches nearly every file. hudi-rs resolves an upsert by its ordering field (the later updated_at wins); delta-rs and PyIceberg overwrite unconditionally. The end states agree because every input is newer than what it replaces.

How it is measured

  • Timed: only the library call: the load, each commit, the full read. Opening the table, reading the commit inputs, correctness checks and file listings are outside the timer.
  • Isolation: each phase (load, upserts, deletes, read) runs in a fresh process, and each of the three repetitions rebuilds the table from scratch.
  • Cleanup: after each commit, outside the timer, the harness deletes data files the table's current snapshot no longer references (--reclaim, on by default), so copy-on-write runs do not keep a copy per commit. It is off on training-set, whose time-travel reads need the old files. No library's own cleaner or vacuum runs.
  • Correctness: every commit's inserted, updated and deleted row counts are checked against the plan, and every end state is compared row by row with a Parquet copy of the expected table. A case that fails either check reports no numbers.
  • Quiet machine: before every timed call the harness checks that other processes kept at most --max-load cores busy over the last 10 seconds (the benchmark's own CPU is left out) and that at least --min-mem-gib of memory is free (measured while the phase holds its data, so it is headroom beyond the benchmark's own use). It waits up to --settle-timeout seconds for load to drop, then refuses: the case is recorded as refused with no numbers. Too little free memory refuses at once; close applications and rerun that case. Both readings are stored next to every timing.
  • Reported numbers: per case, the median over the 3 repetitions of load and read time, and of each repetition's mean upsert and delete commit time. summarize.py prints min to max in parentheses and notes any timing whose variation across repetitions exceeds 10%.

The published numbers were taken on one Apple M3 Max laptop (14 cores, 36 GiB, local SSD), macOS 26.6, with --max-load 6 --min-mem-gib 10 --settle-timeout 1800 --case-timeout 3600. A case whose load, read, or per-repetition commit median varied by more than 10% across repetitions (ignoring under 20 ms), or with any single run over 1.5x its mean, was rerun, and the run with the least variation is reported. That rerun rule is not part of the harness.

Setup

Needs uv (it provides Python 3.13), git, curl and network access (the harness downloads the NYC taxi file and DuckDB's TPC-DS extension). About 20 GB of free disk on a real disk; set UPSERT_BENCH_WORK_DIR if your temp directory is small or memory-backed.

git clone https://github.com/apache/hudi-rs && cd hudi-rs
git checkout 543642771a1440f8470053831463052e190b3923

curl -fsSLO https://gist.githubusercontent.com/yihua/514f656c2adcf9869ef41c265a36948a/raw/d94a68449a1df78d4f36f9b0ff97af865b02c75e/requirements-lock.txt
curl -fsSLO https://gist.githubusercontent.com/yihua/514f656c2adcf9869ef41c265a36948a/raw/d94a68449a1df78d4f36f9b0ff97af865b02c75e/compare_cases.py

benchmark/upsert/run.sh setup
uv pip install --python benchmark/upsert/.venv/bin/python -r requirements-lock.txt

Run

A smoke pass first. Smoke runs use small data, run 3 repetitions each, and only warn on a busy machine:

for s in uuid tpcds-web-returns nyc-taxi rag-chunks training-set; do
  benchmark/upsert/run.sh smoke --scenario "$s" \
    --cases hudi-mor,hudi-cow,hudi-cow-noindex,delta,iceberg --extra-cases compare_cases.py
done

Then full scale, one case per invocation, one at a time. Close other applications first. The limits below are the ones the published numbers used; the harness defaults (--max-load 2 --min-mem-gib 16) are stricter and refuse on most desktops.

mkdir -p results
for s in uuid tpcds-web-returns nyc-taxi rag-chunks training-set; do
  for c in hudi-mor hudi-cow hudi-cow-noindex delta iceberg; do
    benchmark/upsert/run.sh full --scenario "$s" --cases "$c" --repeat 3 \
      --max-load 6 --min-mem-gib 10 --settle-timeout 1800 --case-timeout 3600 \
      --extra-cases compare_cases.py --out "results/$s-$c.json"
  done
done

benchmark/upsert/run.sh summarize results/*.json --markdown

Each result file records the machine, the hudi-rs commit, every library version (deltalake and pyiceberg under each case's config), the exact call behind every operation, and the load and memory readings taken before each timed call. --force turns refusals into warnings, but summarize.py then renders those results only with --allow-loaded.

Known issue: on macOS, PyIceberg 0.12 crashes on the first tpcds-web-returns upsert. pyarrow overflows a worker thread's stack while building the per-key filter for the composite key. The harness records the case as failed. Linux gives threads a larger stack, so the outcome there may differ.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment