Skip to content

Add pyiceberg/ mmh3 non-pure packages + pure package friends - #4648

Merged
hoodmane merged 1 commit into
pyodide:mainfrom
bitsondatadev:add-pyiceberg
Apr 10, 2024
Merged

hoodmane merged 1 commit into
pyodide:mainfrom
bitsondatadev:add-pyiceberg

Conversation

@bitsondatadev

@bitsondatadev bitsondatadev commented Mar 28, 2024 •

Copy link
Copy Markdown
Contributor

Description

This adds the meta.yaml files to support pyIceberg, a Python client for the SQL table storage format for large datasets with periodic dimensional changes. This also includes the meta.yaml files for the mmh3 package.

Related: #4632
Fixes: #4644

Related discussion on setup: #4632

I will be adding tests/docs/etc.. shortly, just wanted to get the PR up for visibility.

Checklists

  • Add a CHANGELOG entry
  • Add / update tests
  • Add new / update outdated documentation

@bitsondatadev
bitsondatadev force-pushed the add-pyiceberg branch 2 times, most recently from 6f6df3a to 4b1cf89 Compare March 29, 2024 22:56
Comment thread packages/pyiceberg/meta.yaml Outdated
summary: Apache Iceberg is an open table format for huge analytic datasets
license: Apache-2.0
requirements:
host:

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

You need to put all these packages in requirements/run. They are not used at build time but at runtime.

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Ah okay, so basically, anything in requirement.txt should go into run and only some special package (say some sort of testing packages) would fall under requirements/host?

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

only some special package

Yes, mostly you don't need to put anything on the requirements/host, except for some special cases such as:

  1. Numpy: numpy is often used in the build time, you can check the pyproject.toml of the project to see if the numpy is defined as a build dependency.
  2. Shared libraries: if the package depends on some shared libraries (non python packages), like openssl, openblas, then you need to place it here.

Comment thread packages/pyiceberg/test_pyiceberg.py Outdated

@run_in_pyodide(
packages=[
"pyiceberg",

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

You can just put pyiceberg here, and remove other packages.

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Hey @ryanking13 / @rth / @hoodmane, I think I'm hitting the same issue here as this issue with these pure python packages:

  'click',
  'fsspec',
  'rich',
  'sortedcontainers', 
  'strictyaml'

I am a bit confused by the outcome or next steps and I don't see anything exactly answering this question in the FAQ.

#1791 (comment)

Thanks :) I'll push what I have shortly.

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

You need to put all those pure packages to requirements/run, then pyodide will load all those dependencies when loading pyiceberg.

Make sure you added meta.yaml for all those pure python packages as well. For now, we require you to add meta.yaml for pure python packages as well as non-pure python packages to make lockfile (There is a related discussion here: #2580).

@bitsondatadev
bitsondatadev marked this pull request as ready for review April 6, 2024 04:56
@bitsondatadev
bitsondatadev requested a review from ryanking13 April 6, 2024 04:56
Comment thread docs/project/changelog.md Resolved
@bitsondatadev bitsondatadev changed the title [DRAFT] Add pyiceberg and mmh3 package Add pyiceberg and mmh3 package Apr 6, 2024

@ryanking13 ryanking13 left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks @bitsondatadev! I have some minor comment otherwise looks good to me.

Comment thread docs/project/changelog.md Resolved
Comment thread packages/rich/meta.yaml Resolved
Comment thread packages/sortedcontainers/meta.yaml Resolved
Comment thread packages/pyiceberg/meta.yaml Resolved
@bitsondatadev

Copy link
Copy Markdown
Contributor Author

Thanks @bitsondatadev! I have some minor comment otherwise looks good to me.

Thanks @ryanking13, just pushed these changes and I will update to better testing for pyodide once #2933 has a resolution. We currently depend on Arrow for local writes. This looks like it's coming along well though so shouldn't be much longer.

@bitsondatadev bitsondatadev changed the title Add pyiceberg and mmh3 package Add pyiceberg/ mmh3 non-pur packages + pure package friends Apr 8, 2024
@bitsondatadev

Copy link
Copy Markdown
Contributor Author

@ryanking13 let me know if there are any other changes needed. Thank you so much for all of the time and explanation you offered!

Comment thread packages/mmh3/meta.yaml Outdated
Comment on lines +13 to +26
'MIT License Copyright (c) 2011-2023 Hajime Senuma Permission is hereby
granted, free of charge, to any person obtaining a copy of this software and associated
documentation files (the "Software"), to deal in the Software without restriction,
including without limitation the rights to use, copy, modify, merge, publish,
distribute, sublicense, and/or sell copies of the Software, and to permit persons
to whom the Software is furnished to do so, subject to the following conditions: The
above copyright notice and this permission notice shall be included in all copies
or substantial portions of the Software. THE SOFTWARE IS PROVIDED "AS IS", WITHOUT
WARRANTY OF ANY KIND, EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES
OF MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO
EVENT SHALL THE AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES
OR OTHER LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING
FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN
THE SOFTWARE.'

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

If this is the normal text of the MIT license, could you shorten it to just license: MIT?

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

lol of course :) this is what came down with the skeleton command and I think all it adds is the author's name, but MIT should be just fine.

@bitsondatadev

Copy link
Copy Markdown
Contributor Author

Done @hoodmane!

@bitsondatadev
bitsondatadev requested a review from hoodmane April 10, 2024 02:04
@bitsondatadev

Copy link
Copy Markdown
Contributor Author

Done @hoodmane!

Oops added another package I was playing with locally

@ryanking13 ryanking13 left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks again!

Comment thread packages/getdaft/meta.yaml Resolved
Comment thread packages/getdaft/test_daft.py Resolved
Comment thread packages/getdaft/test_daft.py Resolved
@hoodmane
hoodmane merged commit 15fe0b1 into pyodide:main Apr 10, 2024
@hoodmane

Copy link
Copy Markdown
Member

Thanks @bitsondatadev!

@bitsondatadev

bitsondatadev commented Apr 10, 2024 •

Copy link
Copy Markdown
Contributor Author

Awesome! Thanks @ryanking13 and @hoodmane! Now we just need to get PyArrow changes in!

@bitsondatadev bitsondatadev changed the title Add pyiceberg/ mmh3 non-pur packages + pure package friends Add pyiceberg/ mmh3 non-pure packages + pure package friends Apr 10, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Add PyIceberg Package

3 participants