Repository navigation
Releases
v0.8.0-rc.1
Release Notes
This is the first release candidate for Daft v0.8.0. For this minor release, we are creating release candidates to give the community a change to test in real workloads and fix any outstanding bugs or performance regressions before the final release. Of the breaking changes, the ones to look out for in particular are:
- Upgrades to our AWS, Azure, and GCS client libraries, which may affect authentication or IO performance in certain edge cases
- Removal of the deprecated
@daft.udfdecorator. We suggest migrating to@daft.func.batchinstead - Removal of the deprecated
daft.io.lancemodule. We suggest migrating to thedaft-lancePython package instead - Changes to the output types from
daft.read_mcap - Other miscellaneous changes
We're really excited to share all the wonderful features and changes the community made in this minor release, it's a big one!
What's Changed
💥 Breaking Changes
- chore!: remove deprecated min_cpu_per_task from ExecutionContext by @Lucas61000 in #7465
- feat(io)!: execute MCAP with native scan tasks by @everettVT in #7339
- chore!: remove deprecated llm_generate by @mikemikimike in #7450
- chore(lance)!: remove deprecated daft.io.lance module by @Lucas61000 in #7514
- chore(logging)!: remove deprecated setup_debug_logger() by @Lucas61000 in #7427
- fix!: Don't use
HTTPConfig.bearer_tokenfor HF auth by @srilman in #7525 - chore!: Upgrade GCS SDK to new package name by @srilman in #7609
- chore!: Remove
DataSourceTask.get_micro_partitionsby @srilman in #7611 - refactor(udf)!: remove legacy @daft.udf API by @hello-peter-tang in #7482
- chore!: Upgrade S3 client library by @srilman in #7622
✨ Features
- feat(mcap): add native MCAP reader core by @everettVT in #7338
- feat(distributed): add bisect autoscaling strategy for faster cluster… by @DogerW666 in #6929
- feat(lance): add lance namespace and insert overwrites mode for write lance by @FANNG1 in #7518
- feat: deltalake metadata pushdowns by @rchowell in #7549
- feat: add Avro file format read/write support by @Lucas61000 in #7009
- feat: migrate Azure IO to GA azure_storage_blob SDK, drop patched fork by @desmondcheongzx in #7368
- feat: adds jev functions by @rchowell in #7558
- feat(sql): support ANSI SUBSTRING syntax by @Anther-off in #7577
- feat(io): add basic ORC file reading support by @jiangxt2 in #7576
- feat(sql): support CROSS JOIN syntax by @easyrev in #7573
- feat(iceberg): read data files without field IDs via schema.name-mapping.default by @YuangGao in #7618
🐛 Bug Fixes
- fix(io): preserve Hugging Face glob revisions by @everettVT in #7322
- fix(ai): forward extra_body to embedding requests by @nyl3532016 in #7516
- fix(sql): report read_iceberg argument errors instead of panicking by @jackylee-ch in #7354
- fix: eliminate TOCTOU race in CREATE TABLE IF NOT EXISTS by @Lucas61000 in #7387
- fix(io): enforce timeouts on OpenDAL clients by @everySympathy in #7523
- fix(iceberg): apply partition predicates to count pushdown by @atovk in #7421
- fix(io): parse container@host Azure URIs on Microsoft Fabric / OneLake hosts by @djouallah in #7533
- fix(io): defer optional PyIceberg imports until validation by @dipeshbabu in #7572
- fix(iceberg): honor write.parquet.compression-codec and compression-level by @YuangGao in #7607
- fix(sql): use signed 64-bit Iceberg snapshot IDs by @Solomon-mithra in #7590
- fix(io): expose documented HTTPConfig properties by @everettVT in #7579
- fix(sql): fix SQL NULL semantics for IS [NOT] TRUE/FALSE by @rchowell in #7620
- fix: Rename
daft.read_mcapparameteruse_legacy_typestooutput_formatby @srilman in #7625 - fix: Bump webpki-roots to 1.0.10 for eMudhra CA compromise by @everettVT in #7628
- fix: handle Ray versions with local suffixes by @dujl in #7654
🚀 Performance
📖 Documentation
- docs: add daft-doris to Community Extensions by @jiangxt2 in #7509
- docs: adds jev docs to nav by @rchowell in #7568
✅ Tests
👷 CI
- ci: Use SeaweedFS instead of MinIO by @srilman in #7550
- ci: Hide Dependency Diff when no changes by @srilman in #7552
- ci: Cache Docker pulls to reduce integration test transient failures by @srilman in #7631
- ci: Fix Nightly Integration Tests by @srilman in #7650
- ci: Fix MacOS transient CI issues by @srilman in #7664
- ci: Fix how pre-release tags are formatted for Cargo by @srilman in #7683
🔧 Maintenance
- chore: Add check for mutable values in default arguments by @srilman in #7494
- chore(deps): upgrade daft-lance to 0.5.0 by @FANNG1 in #7512
- chore(lance): deprecate write_lance merge mode by @FANNG1 in #7526
- chore: Upgrade Rust nightly by @srilman in #7393
- chore: Upgrade azure_storage_blob to 1.1.0 by @srilman in #7610
- chore: Update Python dev dependencies and restrict to <3.15 by @srilman in #7623
New Contributors
- @nyl3532016 made their first contribution in #7516
- @mikemikimike made their first contribution in #7450
- @djouallah made their first contribution in #7533
- @Anther-off made their first contribution in #7577
- @dipeshbabu made their first contribution in #7572
- @easyrev made their first contribution in #7573
- @Solomon-mithra made their first contribution in #7590
Full Changelog: v0.7.25...v0.8.0-rc.1
v0.7.26
Full Changelog: v0.7.25...v0.7.26
Release Notes
This is a smaller release to apply a dependency upgrade before the larger v0.8.0 release.
What's Changed
Build
- build: bump Ray to 2.59 by @NellyWhads in #7567
v0.7.25
What's Changed
✨ Features
- feat(lerobot): support v2.0/v2.1 dataset layouts by @XuQianJin-Stars in #7381
- feat(iceberg): add overwrite_filter for static partition overwrite by @atovk in #7419
- feat(optimizer): push limit through row-preserving join sides by @hello-peter-tang in #7439
- feat(io): add WebDataset reader by @rohankumardubey in #7493
🐛 Bug Fixes
- fix: clean up unscheduled UDF actors by @everettVT in #7346
- fix: IsIn/Decimal/UnresolvedColumn expressions fail to translate to SQL by @jiangxt2 in #7278
- fix(functions): make coalesce() argument validation reachable by @DogerW666 in #7404
- fix(catalog): declare noop auth for Gravitino Iceberg REST catalogs by @kyo-tom in #7417
- fix(distributed): surface background errors while input is pending by @zhouwenjiajia in #7396
- fix(parquet): error instead of nulling columns when field ids are absent by @jackylee-ch in #7446
- fix(catalog): avoid shared mutable default for from_postgres extensions by @DogerW666 in #7426
- fix(sql): escape single quotes in SQL literal rendering by @chuqCTC in #7269
- fix(sql): reject negative literals for unsigned table function options by @jackylee-ch in #7479
- fix(utf8): preserve backslashes in regexp_replace replacement by @jonasdedden in #7472
- fix: Fix startup of IO-Integration Docker Compose by @srilman in #7499
🚀 Performance
♻️ Refactor
- refactor(shuffle): unify distributed shuffle backend config onto daft-local-plan ShuffleBackend by @BABTUNA in #7402
📖 Documentation
- docs(sql): document table function options by @jackylee-ch in #7355
- docs: Update extras installation docs by @srilman in #7391
- docs: remove stale TableState references by @MeteorSkyOne in #7457
✅ Tests
👷 CI
- ci(integration): build Iceberg test image from python:3.10-bookworm by @hello-peter-tang in #7489
New Contributors
- @DogerW666 made their first contribution in #7404
- @atovk made their first contribution in #7419
- @MeteorSkyOne made their first contribution in #7457
- @chuqCTC made their first contribution in #7269
- @yongster made their first contribution in #7481
- @NellyWhads made their first contribution in #7487
- @rohankumardubey made their first contribution in #7493
- @jonasdedden made their first contribution in #7472
Full Changelog: v0.7.24...v0.7.25
v0.7.24
What's Changed
✨ Features
- feat: add read_blob function for reading files as raw bytes by @hello-peter-tang in #7318
- feat(utf8): add split_part string function by @ByJH in #7371
🐛 Bug Fixes
- fix: ls pagination by @prayanshchh in #7372
🔧 Maintenance
- chore: Update Rust deps for dependabot warnings by @srilman in #7363
- chore(deps): bump opendal to 0.58.1 by @Xuanwo in #7358
- chore: Remove some duplicate dependencies by @srilman in #7370
- chore: Bump some dependencies by @srilman in #7378
New Contributors
- @Xuanwo made their first contribution in #7358
- @prayanshchh made their first contribution in #7372
- @hello-peter-tang made their first contribution in #7318
- @ByJH made their first contribution in #7371
Full Changelog: v0.7.23...v0.7.24
v0.7.23
What's Changed
✨ Features
- feat(iceberg): inject DataFile column stats into DataSourceTask for predicate short-circuit by @jiangxt2 in #7301
🐛 Bug Fixes
- fix(dataframe): preserve explain() plan for all write sinks by @desmondcheongzx in #6580
🚀 Performance
📖 Documentation
- docs: migrate tutorials and examples to @daft.func / @daft.cls by @everettVT in #6729
✅ Tests
- test(iceberg): cover read_iceberg against a local catalog by @jackylee-ch in #7357
🔧 Maintenance
- chore: Update dashboard frontend dependencies by @srilman in #7360
- chore: Ignore Dependabot warnings in benchmarks by @srilman in #7361
Full Changelog: v0.7.22...untagged-8f79647338339e50cd69
v0.7.22
What's Changed
✨ Features
- feat(catalog): support Gravitino Iceberg REST catalogs by @kyo-tom in #7245
- feat(io): add HDFS support via OpenDAL services-hdfs by @Lucas61000 in #7202
- feat(io): export HdfsConfig from daft.io by @Lucas61000 in #7298
- feat(sql): support CREATE TABLE statement by @Lucas61000 in #7186
- feat(functions): add try_divide for PySpark parity by @BABTUNA in #7210
- feat(file): trace media file reads by @everettVT in #7312
- feat(file): trace video frame decoding by @everettVT in #7313
- feat(sql): support read_parquet ignore_corrupt_files by @jackylee-ch in #7133
- feat(sql): resolve read_iceberg io_config from table properties and default config by @jackylee-ch in #7273
- feat(sql): support ignore_corrupt_files in read_csv by @jackylee-ch in #7264
🐛 Bug Fixes
- fix(core): handle all-null list literals by @everySympathy in #7168
- fix(parquet): honor row-group constraints in count_rows() count pushdown by @FANNG1 in #7288
- fix: Gravitino catalog S3 credential key mismatch and _has_table error handling by @jiangxt2 in #7279
- fix(io): decode video with a single PyAV generator so .mkv/.webm yield frames by @AnayGarodia in #7229
- fix(io): support writing to hdfs:// URLs that include a port by @Lucas61000 in #7307
- fix(ai): route OpenAI chat completions parse through beta namespace by @colin-ho in #6995
- fix: truncate default IOConfig fields in explain() output by @shreynath in #7327
♻️ Refactor
- refactor(shuffle): migrate repartition onto shared backend and unify pipeline dispatch by @BABTUNA in #7275
- refactor(io): migrate Delta Lake scans to DataSource by @ihopenre-eng in #7300
- refactor(io): migrate Hudi scans to DataSource by @Lucas61000 in #7324
- refactor: migrate IcebergScanOperator to DataSource by @Lucas61000 in #7311
📖 Documentation
- docs: add motion trimming physical AI example by @ykdojo in #7341
- docs: remove dead HDF5 example notebook link by @everettVT in #7352
✅ Tests
- test(ai): mock beta chat completions parse by @everettVT in #7315
New Contributors
- @jiangxt2 made their first contribution in #7279
- @ihopenre-eng made their first contribution in #7300
- @shreynath made their first contribution in #7327
Full Changelog: v0.7.21...v0.7.22
v0.7.21
What's Changed
🐛 Bug Fixes
📖 Documentation
- docs(benchmark): add hand-tracking workload A/B benchmark for lerobot batched decode by @ykdojo in #7267
- docs: add physical AI examples (hand tracking + reward scoring) by @ykdojo in #7271
Full Changelog: v0.7.20...v0.7.21
v0.7.20
What's Changed
✨ Features
- feat(iceberg): push down starts_with and is_nan/not_nan filters by @jackylee-ch in #7266
🐛 Bug Fixes
- fix(dashboard): support OUT_DIR on a different filesystem from the source tree by @alejandro-soto-franco in #7246
- fix(daft-file): release GIL in PyDaftFile methods to prevent deadlock by @zhouwenjiajia in #7257
- fix: fd and thread leak from tqdm progress bar in Jupyter by @ykdojo in #7262
♻️ Refactor
- refactor(shuffle): dedup local shuffle backend enums for gather and into_partitions by @BABTUNA in #7221
🔧 Maintenance
- chore(deps): lower opt-level for opendal-service-oss release builds by @alejandro-soto-franco in #7249
- chore(config): deprecate min_cpu_per_task by @XiaoHongbo-Hope in #7125
New Contributors
- @alejandro-soto-franco made their first contribution in #7249
- @zhouwenjiajia made their first contribution in #7257
Full Changelog: v0.7.19...v0.7.20
v0.7.19
What's Changed
✨ Features
- feat: add mcap pip extra and dependency checks for read_mcap by @everettVT in #7220
- feat(writers): add single_file kwarg to write_parquet by @aaron-ang in #7004
🐛 Bug Fixes
- fix: normalize HF bucket tree paths by @everettVT in #7218
- fix(session): allow set_provider to accept Provider instance for auto… by @DieselZhang in #7041
- fix(telemetry): prevent interpreter-shutdown segfault from Scarf telemetry by @FANNG1 in #7203
- fix(io): route oss:// through S3-compatible storage options for write_lance by @AnayGarodia in #7228
- fix: Reuse file format config in Iceberg and Delta by @srilman in #7163
📖 Documentation
- docs: add GooseFS connector documentation by @XuQianJin-Stars in #7193
- docs: Clean up formatting in Common Crawl docs by @srilman in #7237
- docs(checkpoint): add user guide and connector cross-references by @rohitkulshreshtha in #6985
👷 CI
New Contributors
- @DieselZhang made their first contribution in #7041
- @FANNG1 made their first contribution in #7203
- @AnayGarodia made their first contribution in #7228
Full Changelog: v0.7.18...v0.7.19
v0.7.18
What's Changed 🚀
✨ Features
- feat: Remove dynamodb dependency for concurrent writes in deltalake on s3 @arpatilmh (#7187)
- feat(lerobot): batch video-frame decode by shard @ykdojo (#7184)
- feat(io): export GooseFSConfig from daft.io @XuQianJin-Stars (#7194)
🚀 Performance
Full Changelog: v0.7.17...v0.7.18