Read Iceberg to DeltacatDataset by JonasJ-ap · Pull Request #131 · ray-project/deltacat

JonasJ-ap · 2023-05-10T16:37:59Z

Adds a read_iceberg read API, which uses pyiceberg and Ray Data API read_parquet.

Waiting for `pyiceberg0.4.0" Release (Release Vote Begins)

pdames

As a high-level comment, let's (1) target merging this into a new iceberg branch instead of the main branch for now and (2) ensure that all changes pass the linter by following the steps at https://github.com/ray-project/deltacat/blob/main/README-development.md. You can also review the complete set of suggested changes here that are passing make test-integration on my Macbook at https://github.com/pdames/deltacat/tree/add_read_iceberg.

pdames · 2023-06-01T23:25:38Z

 s3fs == 2022.2.0
 tenacity == 8.1.0
 typing-extensions == 4.4.0
+pyiceberg @ git+https://github.com/apache/iceberg#subdirectory=python


Please add this to install_requires inside of setup.py as well.

pdames · 2023-06-01T23:58:29Z

+install:
+	rm -rf venv
+	python -m venv venv
+	venv/bin/pip install -r dev-requirements.txt
+
+install-apple-silicon:
+	rm -rf venv
+	/usr/bin/python3 -m venv venv
+	venv/bin/pip install -r dev-requirements.txt


While testing on my laptop, I noticed that: (1) always deleting the virtual environment can result in a slow feedback loop of ~5 minutes to recreate venv if it fails late during requirement installation, (2) the pip binary included in venv fails to find modules like ray unless I first upgrade pip via venv/bin/pip install --upgrade pip, and (3) I need to decide which install command to run.

To workaround these issues, I have (1) moved rm -rf venv into a separate clean target that must be explicitly invoked if someone wants a clean install like make clean && make install, (2) always upgraded the venv version of pip via venv/bin/pip install --upgrade pip during each install, and (3) automatically detected whether we're running on arm64 architecture or not. What do you think?

Suggested change

install:

rm -rf venv

python -m venv venv

venv/bin/pip install -r dev-requirements.txt

install-apple-silicon:

rm -rf venv

/usr/bin/python3 -m venv venv

venv/bin/pip install -r dev-requirements.txt

clean:

rm -rf venv

install:

if [ ! -d "venv" ]; then \

if [[ '$(shell uname -m)' == 'arm64' ]]; then \

/usr/bin/python3 -m venv venv; \

else \

python -m venv venv; \

fi \

fi

venv/bin/pip install --upgrade pip

venv/bin/pip install -r dev-requirements.txt

test-integration:

docker-compose -f dev/iceberg-integration/docker-compose-integration.yml kill

docker-compose -f dev/iceberg-integration/docker-compose-integration.yml build

docker-compose -f dev/iceberg-integration/docker-compose-integration.yml up -d

sleep 30

venv/bin/python -m pytest tests/

…ct#131.

Signed-off-by: Rushan Jiang <[email protected]>

…tability when running integration test Signed-off-by: Rushan Jiang <[email protected]>

JonasJ-ap

@pdames Thank you for your review and suggestions. I've incorporated your suggested update and revised the integration test structure.

pdames · 2023-06-28T02:25:57Z

LGTM - nice work @JonasJ-ap!

* add read_iceberg and corresponding test structure * rollback requirements change * prepare for pyiceberg0.4.0 release. Add more tests * add missing requirements * add quick install command * add read_iceberg and corresponding test structure * rollback requirements change * prepare for pyiceberg0.4.0 release. Add more tests * add missing requirements * add quick install command * Fixes for Ray 2.X and add recommendations from review of PR ray-project#131. * fix format issue Signed-off-by: Rushan Jiang <[email protected]> * fix requirements list and setup Signed-off-by: Rushan Jiang <[email protected]> * fix the partitioning issue reported in the ray-project PR Signed-off-by: Rushan Jiang <[email protected]> * porting apache/iceberg#7768 to reduce the waiting time and increase stability when running integration test Signed-off-by: Rushan Jiang <[email protected]> --------- Signed-off-by: Rushan Jiang <[email protected]> Co-authored-by: Patrick Ames <[email protected]>

… and new makefile targets. (#202) * Logging memory consumed to validate worker estimation correctness (#142) * Logging memory consumed to validate worker estimation correctness [Description] This change will allow us to validate the correctness of the workers we estimated to run a compaction job reliably. * update tests * fix tests for python 3.7 as addClassCleanup is absent * addressed comments * Update deltacat/compute/compactor/compaction_session.py Co-authored-by: Patrick Ames <[email protected]> Signed-off-by: Raghavendra M Dani <[email protected]> * Update deltacat/utils/ray_utils/concurrency.py Co-authored-by: Patrick Ames <[email protected]> Signed-off-by: Raghavendra M Dani <[email protected]> * remove dependency on rebase_source_partition, simplify the repartition delta discovery * remove repartition_during_rebase argument, ping the source to compacted table * remove repartition rcf write, generalize the rcf write by supporting pre-specified url * addressed comments * logging cpu usage metrics * fix percentages * capture latency of retrieving cluster resources --------- Signed-off-by: Raghavendra M Dani <[email protected]> Co-authored-by: Patrick Ames <[email protected]> Co-authored-by: Jialin Liu <[email protected]> * Capturing all the performance metrics in an audit (#146) * Capturing all the performance metrics in an audit [Description] It is important to capture audit automatically and analyze the datapoints from significant number of runs to recognize a pattern and come up with a formula to predict resource consumption. This would improve reliability and efficiency. [Motivation] #141 * fix unit tests * refactoring and captured new metrics * fix constructor args * fix assert statement * fix NoneType error * fix bug in skipping manifest * add comment and address comments * fix stage_delta * fixing deltacat method signature * capturing the head process peak memory usage and result size of each step * [skip download cold manifest] Add support for skipping download cold manifest entries during materialize step * Address review comments * Adding a data model to represent compact_partition function parameters (#151) * defined CompactPartitionParams class * fixed setuptools.setup name * updated CompactPartitionParams docstring to refer to function location * added compact_partition_from_request method that wraps compact_partition * fixed unit tests * fixed test_serialize_handles_sets unit test by asserting that two list contain the same elements instead of using == operator * fixed linting in test_compact_partition_params.py * Object store implementation to allow elastically increasing the object store memory (#149) * [skip untouched files]Disable copy by reference during backfill and rebase only compaction; Fix referenced manifest PyarrowWriteResult (#153) * add dd parallelism params (#154) Co-authored-by: Jialin Liu <[email protected]> * Allow s3 client kwargs as argument of compact_partition (#155) * Allow s3 client kwargs as argument of compact_partition * Changing default as empty dict * add s3_client_kwargs to rcf * Bunping up version to 0.1.18b8 * Honor profile name in s3 client kwargs (#157) * Use kwargs as to determine the profile to use when creating the s3 client * bumped up version to 0.1.18b9 * Allow s3_client_kwargs to be passed into repartition (#158) * Allow s3_client_kwargs to be passed into repartition * Update compaction session s3_client_kwargs to default to None * Move s3_client_kwargs default setter to parent scope (#159) * Move s3_client_kwargs default setter to parent scope * Bump version to 0.1.18b11 * keep null row and remove dw_int64 column (#161) * keep null row and remove dw_int64 column * add assertation for extra column remove * remove column by name explicitly * use drop column * use list of str in drop columns --------- Co-authored-by: Jialin Liu <[email protected]> * version bump to 0.1.18b12 (#164) * Cleaning up rehashing logic as it is a dead code as of now. (#166) * Cleaning up rehashing logic as it is a dead code as of now. [Motivation] #165 * fix version number * Fix stream position and support latest pyarrow (#168) [Motivation] #167 * Bumped version from 0.1.18b12 to 0.1.18b13 (#169) * bumped version from 0.1.18b12 to 0.1.18b13 * fixed current_node_name logic to handle ipv6 addresses * Add pytest benchmarking for Parquet reads (#160) * Add benchmarking scripts and README * Cleanup code and report visualizations * Enable 1RG tpch files * Fix naming from 1RG to 2RG * Update dev-requirements.txt Co-authored-by: Patrick Ames <[email protected]> Signed-off-by: Jay Chia <[email protected]> * Update deltacat/benchmarking/conftest.py Co-authored-by: Patrick Ames <[email protected]> Signed-off-by: Jay Chia <[email protected]> * Update deltacat/benchmarking/README.md Co-authored-by: Patrick Ames <[email protected]> Signed-off-by: Jay Chia <[email protected]> * Lints and README * Use benchmark-requirements.txt instead of dev-requirements.txt * Lints --------- Signed-off-by: Jay Chia <[email protected]> Co-authored-by: Jay Chia <[email protected]@users.noreply.github.com> Co-authored-by: Patrick Ames <[email protected]> * Polling EC2 Instance Metadata endpoint until HTTP 200 OK (#172) * block_until_instance_metadata_service_returns_success implemented * now just checking INSTANCE_METADATA_SERVICE_IPV4_URI * added unit testing * removed unneccesary logging * removed commented out print statement * fixed session.mount * removed unused args * now using tenacity for retrying * changed default stop strategy * removed unused import in TestBlockUntilInstanceMetadataServiceReturnsSuccess * retry_url->retrying_get, log_attempt_number now private * Adding local deltacat storage module (#175) * Adding local deltacat storage module * Using camel case dict keys for consistency * Support storing data in parquet and utsv bytes * Fix README.md to allow db_file_path arg * version bump from 0.1.18b13 to 0.1.18b14 (#179) * Now triggering publish-to-pypi on editing and creating a release (#180) * `compact_partition` incremental unit test (#188) * fixed commit_partition and list_deltas local deltacat storage bug * providing destination table with deltas * ds_mock_kwargs fixtures just consist of dict of db_file_path * working test_compact_partition_success unit test * flake8 issues * removed print statements * fixed assert ( manifest_records == len(compacted_table) * 3 working unit test cases * implemented test_compact_partition_incremental * commented out broken unit test * removed unused test files * added explanatory comment to block_until_instance_metadata_service_returns_success * reverting ray_utils/runtime change * fixed test_retrying_on_statuses_in_status_force_list * fixed test_retrying_on_statuses_in_status_force_list * added additional block_until_instance_metadata_service_returns_success unit test * additional fixtures * validation_callback_func_ -> validation_callback_func_kwargs * added use_prev_compacted key * added additional use_prev_compacted * fixed test_compaction_session unit tests * copied over working unit tests from dev/test_compact_partition_first_cut * fixed repartition unit tests * moved test_utils to itw own module * removed unused kwargs arg * paramtrizing records_per_compacted_file, hash_bucket_count * removed unused TODO * augmented CompactPartitionParams to include additional compact_partition kwargs * augmented CompactPartitionParams to include additional compact_partition kwargs * refactored testcases and setup to there own modules * added additional type hints * defaulting to empty dict safetly wherever deltacat_storage_kwargs is a param * revert change that no longer passed **list_deltas_kwargs to io.discover_deltas * additional type hints * no longer passing kwargs to dedupe + decimal-pk case * no longer passing kwargs to materialize * unpacking deltacat_storage_kwargs in deltacat_storage.stage_delta timed_invocation * no longer unpacking **list_deltas_kwargs when calling _execute_compaction_round * readded kwargs * added incremental timestamp-pk unit test * added docstring to offer_iso8601_timestamp_list * added tc for duplicate w sorting keyand multiple primary keys * removed deadcode * added additional comments to first incremental test case * fixture refactoring - compaction_artifacts_s3_bucket -> setup_compaction_artifacts_s3_bucket * parameterizing table version * # ds_mock_kwargs -> # teardown_local_deltacat_storage_db * tearing down db within test execution is now parameterized * added INCREMENTAL_DEPENDENT_TEST_CASES dependent test case * reversing order of sk in 10-incremental-decimal-pk-multi-dup * Switch botocore retry mode to adaptive from standard (#191) Co-authored-by: Kevin Yan <[email protected]> * Merge phash_main into main branch (#195) * Adding support for reading table into ParquetFile (#163) * Adding support for reading just the parquet meta by leveraging ParquetFile [Description] Right now we read the entire table using read_parquet method of pyarrow. However, we lose the parquet metadata in the process and this will eagerly load all the data. In this commit we will use ParquetFile (https://arrow.apache.org/docs/python/generated/pyarrow.parquet.ParquetFile.html) [Motivation] #162 * Using pyarrow.fs.S3FileSystem instead of s3fs * Adding s3 adaptive retries and validate encoding * Interface for merge v2 (#182) * Adding interface and definitions for merge step * fix tests and merge logs * Add hashed memcached client support (#173) * Adding support for reading table into ParquetFile (#163) * Adding support for reading just the parquet meta by leveraging ParquetFile [Description] Right now we read the entire table using read_parquet method of pyarrow. However, we lose the parquet metadata in the process and this will eagerly load all the data. In this commit we will use ParquetFile (https://arrow.apache.org/docs/python/generated/pyarrow.parquet.ParquetFile.html) [Motivation] #162 * Using pyarrow.fs.S3FileSystem instead of s3fs * Adding s3 adaptive retries and validate encoding * Add hashed memcached client support * Adding support for reading table into ParquetFile (#163) * Adding support for reading just the parquet meta by leveraging ParquetFile [Description] Right now we read the entire table using read_parquet method of pyarrow. However, we lose the parquet metadata in the process and this will eagerly load all the data. In this commit we will use ParquetFile (https://arrow.apache.org/docs/python/generated/pyarrow.parquet.ParquetFile.html) [Motivation] #162 * Using pyarrow.fs.S3FileSystem instead of s3fs * Adding s3 adaptive retries and validate encoding * use uid instead of ip address as key to hash algorithm --------- Co-authored-by: Raghavendra M Dani <[email protected]> * Implementing hash bucketing v2 (#178) * Implementing hash bucket v2 * Fix the assertion regarding hash buckets * Python 3.7 does not have doClassCleanups in super * Fix the memory issue with the hb index creation * Compaction session implementation for algo v2 (#187) * Compaction session implementation for algo v2 * Address comments * Added capability to measure instance minutes in a autoscaling cluster setting * Avoid configuring logger if ray is uninitialized * Add readme to run tests * Refactoring and fixing the num_cpus key in options * Resolve merge conflict and rebase from main * Adding additional optimization from POC (#194) * Adding additional optimization from POC * fix typo * Moved the compact_partition tests to top level module * Adding unit tests for parquet downloaders * fix typo * fix repartition session * Adding stack trace and passing config kwargs separate due to s3fs bug * fix the parquet reader * pass deltacat_storage_kwargs in repartition_session * addressed comments and extend tests to handle v2 --------- Co-authored-by: Zyiqin-Miranda <[email protected]> * Daft Native Reader for Parquet Content Types (#183) * Add pyarrow_to_daft_schema helper * added initial daft read for parquet * integrate io code with daft cast * suggestions * lint-fix EOF for requirements * lint-fixs EOF with pre-commit * bump version to 0.1.18b15 * Fix tuple issue with coerce_int96_timestamp_unit * Fix kwargs and Schema * Lint * using include column names instead of column names * Update to use new Daft Schema.from_pyarrow_schema method * Remove daft requirement from benchmark-requirements.txt * Factor out daft parquet reading code into seperate util file * add timing to daft.read_parquet, address feedback, add unit tests * run pre-commit * switch on columns and add local file * add schema override unit test * add schema override unit test * fix fstrings * thread column names through * downgrade to daft==0.1.12 * add support for partial row group downloads * add `reader_type` to pyarrow utils --------- Co-authored-by: Jay Chia <[email protected]@users.noreply.github.com> * [WIP] Read Iceberg to DeltaCAT Dataset (#131) * add read_iceberg and corresponding test structure * rollback requirements change * prepare for pyiceberg0.4.0 release. Add more tests * add missing requirements * add quick install command * add read_iceberg and corresponding test structure * rollback requirements change * prepare for pyiceberg0.4.0 release. Add more tests * add missing requirements * add quick install command * Fixes for Ray 2.X and add recommendations from review of PR #131. * fix format issue Signed-off-by: Rushan Jiang <[email protected]> * fix requirements list and setup Signed-off-by: Rushan Jiang <[email protected]> * fix the partitioning issue reported in the ray-project PR Signed-off-by: Rushan Jiang <[email protected]> * porting apache/iceberg#7768 to reduce the waiting time and increase stability when running integration test Signed-off-by: Rushan Jiang <[email protected]> --------- Signed-off-by: Rushan Jiang <[email protected]> Co-authored-by: Patrick Ames <[email protected]> * Add workaround for pydantic 2.0 incompatibility, add build & deploy to s3, and add AWS glue job runner. * Fix worker logging on AWS Glue, stop duplicate pip installs of DeltaCAT, linting. * Add regionalization, remove assumptions about high-level errors/responses from Glue. --------- Signed-off-by: Raghavendra M Dani <[email protected]> Signed-off-by: Jay Chia <[email protected]> Signed-off-by: Rushan Jiang <[email protected]> Signed-off-by: Patrick Ames <[email protected]> Co-authored-by: jialin <[email protected]> Co-authored-by: Raghavendra M Dani <[email protected]> Co-authored-by: Jialin Liu <[email protected]> Co-authored-by: Zyiqin-Miranda <[email protected]> Co-authored-by: zyiqin <[email protected]> Co-authored-by: pf <[email protected]> Co-authored-by: rkenmi <[email protected]> Co-authored-by: Jay Chia <[email protected]> Co-authored-by: Jay Chia <[email protected]@users.noreply.github.com> Co-authored-by: Kevin Yan <[email protected]> Co-authored-by: Kevin Yan <[email protected]> Co-authored-by: Sammy Sidhu <[email protected]> Co-authored-by: Jonas(Rushan) Jiang <[email protected]>

* add read_iceberg and corresponding test structure * rollback requirements change * prepare for pyiceberg0.4.0 release. Add more tests * add missing requirements * add quick install command * add read_iceberg and corresponding test structure * rollback requirements change * prepare for pyiceberg0.4.0 release. Add more tests * add missing requirements * add quick install command * Fixes for Ray 2.X and add recommendations from review of PR ray-project#131. * fix format issue Signed-off-by: Rushan Jiang <[email protected]> * fix requirements list and setup Signed-off-by: Rushan Jiang <[email protected]> * fix the partitioning issue reported in the ray-project PR Signed-off-by: Rushan Jiang <[email protected]> * porting apache/iceberg#7768 to reduce the waiting time and increase stability when running integration test Signed-off-by: Rushan Jiang <[email protected]> --------- Signed-off-by: Rushan Jiang <[email protected]> Co-authored-by: Patrick Ames <[email protected]>

* [WIP] Read Iceberg to DeltaCAT Dataset (#131) * add read_iceberg and corresponding test structure * rollback requirements change * prepare for pyiceberg0.4.0 release. Add more tests * add missing requirements * add quick install command * add read_iceberg and corresponding test structure * rollback requirements change * prepare for pyiceberg0.4.0 release. Add more tests * add missing requirements * add quick install command * Fixes for Ray 2.X and add recommendations from review of PR #131. * fix format issue Signed-off-by: Rushan Jiang <[email protected]> * fix requirements list and setup Signed-off-by: Rushan Jiang <[email protected]> * fix the partitioning issue reported in the ray-project PR Signed-off-by: Rushan Jiang <[email protected]> * porting apache/iceberg#7768 to reduce the waiting time and increase stability when running integration test Signed-off-by: Rushan Jiang <[email protected]> --------- Signed-off-by: Rushan Jiang <[email protected]> Co-authored-by: Patrick Ames <[email protected]> * Add workaround for pydantic 2.0 incompatibility, add build & deploy to s3, and add AWS glue job runner. * Fix worker logging on AWS Glue, stop duplicate pip installs of DeltaCAT, linting. * Add regionalization, remove assumptions about high-level errors/responses from Glue. * [WIP] First working version of Iceberg bucketed partition writeback using Daft. * Update dependency versions. * Initial draft of metadata model changes for iceberg support. * Second draft of metadata model changes for Iceberg support. * [WIP] DeltaCAT intermediate durable storage model. * [WIP] DeltaCAT durable storage model - all tests passing. * [WIP] DeltaCAT durable storage model w/ metafile base class. * Minor README updates. Linter updates. * Add logo to readme. Update metafile to work with any pyarrow filesystem. * Add durable SerDe to table version schemas. * Add benchmark requirements to make install and fix broken Partition static factory. * Revert s3u and remove dev autoscaler config. * Remove deprecated DeltaCAT datasource and read API extensions. --------- Signed-off-by: Rushan Jiang <[email protected]> Co-authored-by: Jonas(Rushan) Jiang <[email protected]>

* [WIP] Read Iceberg to DeltaCAT Dataset (ray-project#131) * add read_iceberg and corresponding test structure * rollback requirements change * prepare for pyiceberg0.4.0 release. Add more tests * add missing requirements * add quick install command * add read_iceberg and corresponding test structure * rollback requirements change * prepare for pyiceberg0.4.0 release. Add more tests * add missing requirements * add quick install command * Fixes for Ray 2.X and add recommendations from review of PR ray-project#131. * fix format issue Signed-off-by: Rushan Jiang <[email protected]> * fix requirements list and setup Signed-off-by: Rushan Jiang <[email protected]> * fix the partitioning issue reported in the ray-project PR Signed-off-by: Rushan Jiang <[email protected]> * porting apache/iceberg#7768 to reduce the waiting time and increase stability when running integration test Signed-off-by: Rushan Jiang <[email protected]> --------- Signed-off-by: Rushan Jiang <[email protected]> Co-authored-by: Patrick Ames <[email protected]> * Add workaround for pydantic 2.0 incompatibility, add build & deploy to s3, and add AWS glue job runner. * Fix worker logging on AWS Glue, stop duplicate pip installs of DeltaCAT, linting. * Add regionalization, remove assumptions about high-level errors/responses from Glue. * [WIP] First working version of Iceberg bucketed partition writeback using Daft. * Update dependency versions. * Initial draft of metadata model changes for iceberg support. * Second draft of metadata model changes for Iceberg support. * [WIP] DeltaCAT intermediate durable storage model. * [WIP] DeltaCAT durable storage model - all tests passing. * [WIP] DeltaCAT durable storage model w/ metafile base class. * Minor README updates. Linter updates. * Add logo to readme. Update metafile to work with any pyarrow filesystem. * Add durable SerDe to table version schemas. * Add benchmark requirements to make install and fix broken Partition static factory. * Revert s3u and remove dev autoscaler config. * Remove deprecated DeltaCAT datasource and read API extensions. --------- Signed-off-by: Rushan Jiang <[email protected]> Co-authored-by: Jonas(Rushan) Jiang <[email protected]>

JonasJ-ap added 2 commits May 9, 2023 22:12

add read_iceberg and corresponding test structure

fcab194

rollback requirements change

1873472

JonasJ-ap commented May 10, 2023

View reviewed changes

Comment thread setup.py Resolved

pdames self-requested a review May 10, 2023 17:06

JonasJ-ap added 3 commits May 24, 2023 09:12

prepare for pyiceberg0.4.0 release. Add more tests

b95fb38

add missing requirements

51087dd

add quick install command

40ed924

JonasJ-ap marked this pull request as ready for review May 24, 2023 16:27

pdames added a commit to pdames/deltacat that referenced this pull request Jun 25, 2023

Fixes for Ray 2.X and add recommendations from review of PR ray-proje…

ffea239

…ct#131.

pdames requested changes Jun 25, 2023

View reviewed changes

JonasJ-ap and others added 6 commits June 24, 2023 19:13

add read_iceberg and corresponding test structure

98f26bf

rollback requirements change

e824faa

prepare for pyiceberg0.4.0 release. Add more tests

319cdb1

add missing requirements

29df87f

add quick install command

224c9ac

Fixes for Ray 2.X and add recommendations from review of PR ray-proje…

343a45b

…ct#131.

pdames mentioned this pull request Jun 26, 2023

Support read Iceberg dataset in Ray in distributed mode #116

Closed

JonasJ-ap changed the base branch from main to iceberg June 27, 2023 20:53

JonasJ-ap and others added 5 commits June 27, 2023 14:37

merge from Patrick's branch:343a45b7bfa077cdd0ff19d72681b61b77e4213e

7a5bbbf

Signed-off-by: Rushan Jiang <[email protected]>

fix format issue

9601012

Signed-off-by: Rushan Jiang <[email protected]>

fix requirements list and setup

bc7c955

Signed-off-by: Rushan Jiang <[email protected]>

fix the partitioning issue reported in the ray-project PR

152f733

Signed-off-by: Rushan Jiang <[email protected]>

porting apache/iceberg#7768 to reduce the waiting time and increase s…

01814dd

…tability when running integration test Signed-off-by: Rushan Jiang <[email protected]>

JonasJ-ap commented Jun 27, 2023

View reviewed changes

pdames merged commit d0611bf into ray-project:iceberg Jun 28, 2023

JonasJ-ap changed the title ~~[WIP] Read Iceberg to DeltacatDataset~~ Read Iceberg to DeltacatDataset Jun 30, 2023

JonasJ-ap mentioned this pull request Jun 30, 2023

[WIP][Data] Add read_iceberg API to support scalable iceberg scans to ray datasets ray-project/ray#34236

Closed

1 of 8 tasks

Navigation Menu

Repository navigation

Read Iceberg to DeltacatDataset - #131

Read Iceberg to DeltacatDataset#131
pdames merged 16 commits into
ray-project:icebergfrom
JonasJ-ap:add_read_iceberg

JonasJ-ap commented May 10, 2023 •

edited

Loading

Uh oh!

Uh oh!

pdames left a comment

Uh oh!

Uh oh!

pdames Jun 1, 2023

Uh oh!

pdames Jun 1, 2023

Uh oh!

Uh oh!

JonasJ-ap left a comment

Uh oh!

pdames commented Jun 28, 2023

Uh oh!

Reviewers

Uh oh!

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

Conversation

JonasJ-ap commented May 10, 2023 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Uh oh!

pdames left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

pdames Jun 1, 2023

Choose a reason for hiding this comment

Uh oh!

pdames Jun 1, 2023

Choose a reason for hiding this comment

Uh oh!

Uh oh!

JonasJ-ap left a comment

Choose a reason for hiding this comment

Uh oh!

pdames commented Jun 28, 2023

Uh oh!

Reviewers

Uh oh!

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

JonasJ-ap commented May 10, 2023 •

edited

Loading