Skip to content

Releases

v0.8.0-rc.1

v0.8.0-rc.1 Pre-release
Pre-release

Choose a tag to compare

@github-actions github-actions released this 04 Nov 00:09
a037380

Release Notes

This is the first release candidate for Daft v0.8.0. For this minor release, we are creating release candidates to give the community a change to test in real workloads and fix any outstanding bugs or performance regressions before the final release. Of the breaking changes, the ones to look out for in particular are:

  • Upgrades to our AWS, Azure, and GCS client libraries, which may affect authentication or IO performance in certain edge cases
  • Removal of the deprecated @daft.udf decorator. We suggest migrating to @daft.func.batch instead
  • Removal of the deprecated daft.io.lance module. We suggest migrating to the daft-lance Python package instead
  • Changes to the output types from daft.read_mcap
  • Other miscellaneous changes

We're really excited to share all the wonderful features and changes the community made in this minor release, it's a big one!

What's Changed

💥 Breaking Changes

✨ Features

  • feat(mcap): add native MCAP reader core by @everettVT in #7338
  • feat(distributed): add bisect autoscaling strategy for faster cluster… by @DogerW666 in #6929
  • feat(lance): add lance namespace and insert overwrites mode for write lance by @FANNG1 in #7518
  • feat: deltalake metadata pushdowns by @rchowell in #7549
  • feat: add Avro file format read/write support by @Lucas61000 in #7009
  • feat: migrate Azure IO to GA azure_storage_blob SDK, drop patched fork by @desmondcheongzx in #7368
  • feat: adds jev functions by @rchowell in #7558
  • feat(sql): support ANSI SUBSTRING syntax by @Anther-off in #7577
  • feat(io): add basic ORC file reading support by @jiangxt2 in #7576
  • feat(sql): support CROSS JOIN syntax by @easyrev in #7573
  • feat(iceberg): read data files without field IDs via schema.name-mapping.default by @YuangGao in #7618

🐛 Bug Fixes

  • fix(io): preserve Hugging Face glob revisions by @everettVT in #7322
  • fix(ai): forward extra_body to embedding requests by @nyl3532016 in #7516
  • fix(sql): report read_iceberg argument errors instead of panicking by @jackylee-ch in #7354
  • fix: eliminate TOCTOU race in CREATE TABLE IF NOT EXISTS by @Lucas61000 in #7387
  • fix(io): enforce timeouts on OpenDAL clients by @everySympathy in #7523
  • fix(iceberg): apply partition predicates to count pushdown by @atovk in #7421
  • fix(io): parse container@host Azure URIs on Microsoft Fabric / OneLake hosts by @djouallah in #7533
  • fix(io): defer optional PyIceberg imports until validation by @dipeshbabu in #7572
  • fix(iceberg): honor write.parquet.compression-codec and compression-level by @YuangGao in #7607
  • fix(sql): use signed 64-bit Iceberg snapshot IDs by @Solomon-mithra in #7590
  • fix(io): expose documented HTTPConfig properties by @everettVT in #7579
  • fix(sql): fix SQL NULL semantics for IS [NOT] TRUE/FALSE by @rchowell in #7620
  • fix: Rename daft.read_mcap parameter use_legacy_types to output_format by @srilman in #7625
  • fix: Bump webpki-roots to 1.0.10 for eMudhra CA compromise by @everettVT in #7628
  • fix: handle Ray versions with local suffixes by @dujl in #7654

🚀 Performance

  • perf(core): reduce temporary allocations in fixed-size list take by @ConeyLiu in #7566

📖 Documentation

✅ Tests

👷 CI

  • ci: Use SeaweedFS instead of MinIO by @srilman in #7550
  • ci: Hide Dependency Diff when no changes by @srilman in #7552
  • ci: Cache Docker pulls to reduce integration test transient failures by @srilman in #7631
  • ci: Fix Nightly Integration Tests by @srilman in #7650
  • ci: Fix MacOS transient CI issues by @srilman in #7664
  • ci: Fix how pre-release tags are formatted for Cargo by @srilman in #7683

🔧 Maintenance

  • chore: Add check for mutable values in default arguments by @srilman in #7494
  • chore(deps): upgrade daft-lance to 0.5.0 by @FANNG1 in #7512
  • chore(lance): deprecate write_lance merge mode by @FANNG1 in #7526
  • chore: Upgrade Rust nightly by @srilman in #7393
  • chore: Upgrade azure_storage_blob to 1.1.0 by @srilman in #7610
  • chore: Update Python dev dependencies and restrict to <3.15 by @srilman in #7623

New Contributors

Full Changelog: v0.7.25...v0.8.0-rc.1

v0.7.26

Choose a tag to compare

@srilman srilman released this 06 Oct 16:36

Full Changelog: v0.7.25...v0.7.26

Release Notes

This is a smaller release to apply a dependency upgrade before the larger v0.8.0 release.

What's Changed

Build

v0.7.25

Choose a tag to compare

@github-actions github-actions released this 11 Sep 23:48
50b3f12

What's Changed

✨ Features

🐛 Bug Fixes

  • fix: clean up unscheduled UDF actors by @everettVT in #7346
  • fix: IsIn/Decimal/UnresolvedColumn expressions fail to translate to SQL by @jiangxt2 in #7278
  • fix(functions): make coalesce() argument validation reachable by @DogerW666 in #7404
  • fix(catalog): declare noop auth for Gravitino Iceberg REST catalogs by @kyo-tom in #7417
  • fix(distributed): surface background errors while input is pending by @zhouwenjiajia in #7396
  • fix(parquet): error instead of nulling columns when field ids are absent by @jackylee-ch in #7446
  • fix(catalog): avoid shared mutable default for from_postgres extensions by @DogerW666 in #7426
  • fix(sql): escape single quotes in SQL literal rendering by @chuqCTC in #7269
  • fix(sql): reject negative literals for unsigned table function options by @jackylee-ch in #7479
  • fix(utf8): preserve backslashes in regexp_replace replacement by @jonasdedden in #7472
  • fix: Fix startup of IO-Integration Docker Compose by @srilman in #7499

🚀 Performance

♻️ Refactor

  • refactor(shuffle): unify distributed shuffle backend config onto daft-local-plan ShuffleBackend by @BABTUNA in #7402

📖 Documentation

✅ Tests

  • test: use runtime concurrency in morsel plan assertions by @ConeyLiu in #7454

👷 CI

New Contributors

Full Changelog: v0.7.24...v0.7.25

v0.7.24

Choose a tag to compare

@github-actions github-actions released this 14 Aug 22:18
9c2b73e

What's Changed

✨ Features

🐛 Bug Fixes

🔧 Maintenance

New Contributors

Full Changelog: v0.7.23...v0.7.24

v0.7.23

Choose a tag to compare

@github-actions github-actions released this 05 Aug 19:44
1d287b5

What's Changed

✨ Features

  • feat(iceberg): inject DataFile column stats into DataSourceTask for predicate short-circuit by @jiangxt2 in #7301

🐛 Bug Fixes

🚀 Performance

  • perf: Optimize stats aggregation within Swordfish by @srilman in #7359

📖 Documentation

  • docs: migrate tutorials and examples to @daft.func / @daft.cls by @everettVT in #6729

✅ Tests

🔧 Maintenance

  • chore: Update dashboard frontend dependencies by @srilman in #7360
  • chore: Ignore Dependabot warnings in benchmarks by @srilman in #7361

Full Changelog: v0.7.22...untagged-8f79647338339e50cd69

v0.7.22

Choose a tag to compare

@github-actions github-actions released this 03 Aug 22:57
b5d58ed

What's Changed

✨ Features

🐛 Bug Fixes

  • fix(core): handle all-null list literals by @everySympathy in #7168
  • fix(parquet): honor row-group constraints in count_rows() count pushdown by @FANNG1 in #7288
  • fix: Gravitino catalog S3 credential key mismatch and _has_table error handling by @jiangxt2 in #7279
  • fix(io): decode video with a single PyAV generator so .mkv/.webm yield frames by @AnayGarodia in #7229
  • fix(io): support writing to hdfs:// URLs that include a port by @Lucas61000 in #7307
  • fix(ai): route OpenAI chat completions parse through beta namespace by @colin-ho in #6995
  • fix: truncate default IOConfig fields in explain() output by @shreynath in #7327

♻️ Refactor

  • refactor(shuffle): migrate repartition onto shared backend and unify pipeline dispatch by @BABTUNA in #7275
  • refactor(io): migrate Delta Lake scans to DataSource by @ihopenre-eng in #7300
  • refactor(io): migrate Hudi scans to DataSource by @Lucas61000 in #7324
  • refactor: migrate IcebergScanOperator to DataSource by @Lucas61000 in #7311

📖 Documentation

  • docs: add motion trimming physical AI example by @ykdojo in #7341
  • docs: remove dead HDF5 example notebook link by @everettVT in #7352

✅ Tests

New Contributors

Full Changelog: v0.7.21...v0.7.22

v0.7.21

Choose a tag to compare

@github-actions github-actions released this 17 Jul 18:32
7e52cc9

What's Changed

🐛 Bug Fixes

  • fix(ai): keep UDF options out of OpenAI requests by @FANNG1 in #7277

📖 Documentation

  • docs(benchmark): add hand-tracking workload A/B benchmark for lerobot batched decode by @ykdojo in #7267
  • docs: add physical AI examples (hand tracking + reward scoring) by @ykdojo in #7271

Full Changelog: v0.7.20...v0.7.21

v0.7.20

Choose a tag to compare

@github-actions github-actions released this 14 Jul 23:12
55290cb

What's Changed

✨ Features

  • feat(iceberg): push down starts_with and is_nan/not_nan filters by @jackylee-ch in #7266

🐛 Bug Fixes

  • fix(dashboard): support OUT_DIR on a different filesystem from the source tree by @alejandro-soto-franco in #7246
  • fix(daft-file): release GIL in PyDaftFile methods to prevent deadlock by @zhouwenjiajia in #7257
  • fix: fd and thread leak from tqdm progress bar in Jupyter by @ykdojo in #7262

♻️ Refactor

  • refactor(shuffle): dedup local shuffle backend enums for gather and into_partitions by @BABTUNA in #7221

🔧 Maintenance

New Contributors

Full Changelog: v0.7.19...v0.7.20

v0.7.19

Choose a tag to compare

@github-actions github-actions released this 10 Jul 00:16
cf35023

What's Changed

✨ Features

  • feat: add mcap pip extra and dependency checks for read_mcap by @everettVT in #7220
  • feat(writers): add single_file kwarg to write_parquet by @aaron-ang in #7004

🐛 Bug Fixes

  • fix: normalize HF bucket tree paths by @everettVT in #7218
  • fix(session): allow set_provider to accept Provider instance for auto… by @DieselZhang in #7041
  • fix(telemetry): prevent interpreter-shutdown segfault from Scarf telemetry by @FANNG1 in #7203
  • fix(io): route oss:// through S3-compatible storage options for write_lance by @AnayGarodia in #7228
  • fix: Reuse file format config in Iceberg and Delta by @srilman in #7163

📖 Documentation

👷 CI

  • ci: Include New Contributers in Automated Draft Releases by @srilman in #7234

New Contributors

Full Changelog: v0.7.18...v0.7.19

v0.7.18

Choose a tag to compare

@github-actions github-actions released this 08 Jul 23:16
12a95e7

What's Changed 🚀

✨ Features

🚀 Performance

  • perf(inline-agg): pack two-string-column keys into u64 for typed FNV grouping @BABTUNA (#6924)

Full Changelog: v0.7.17...v0.7.18