Repository navigation
Add pyiceberg/ mmh3 non-pure packages + pure package friends - #4648
Conversation
6f6df3a to
4b1cf89
Compare
| summary: Apache Iceberg is an open table format for huge analytic datasets | ||
| license: Apache-2.0 | ||
| requirements: | ||
| host: |
There was a problem hiding this comment.
You need to put all these packages in requirements/run. They are not used at build time but at runtime.
There was a problem hiding this comment.
Ah okay, so basically, anything in requirement.txt should go into run and only some special package (say some sort of testing packages) would fall under requirements/host?
There was a problem hiding this comment.
only some special package
Yes, mostly you don't need to put anything on the requirements/host, except for some special cases such as:
- Numpy: numpy is often used in the build time, you can check the
pyproject.tomlof the project to see if the numpy is defined as a build dependency. - Shared libraries: if the package depends on some shared libraries (non python packages), like openssl, openblas, then you need to place it here.
|
|
||
| @run_in_pyodide( | ||
| packages=[ | ||
| "pyiceberg", |
There was a problem hiding this comment.
You can just put pyiceberg here, and remove other packages.
There was a problem hiding this comment.
Hey @ryanking13 / @rth / @hoodmane, I think I'm hitting the same issue here as this issue with these pure python packages:
'click',
'fsspec',
'rich',
'sortedcontainers',
'strictyaml'
I am a bit confused by the outcome or next steps and I don't see anything exactly answering this question in the FAQ.
Thanks :) I'll push what I have shortly.
There was a problem hiding this comment.
You need to put all those pure packages to requirements/run, then pyodide will load all those dependencies when loading pyiceberg.
Make sure you added meta.yaml for all those pure python packages as well. For now, we require you to add meta.yaml for pure python packages as well as non-pure python packages to make lockfile (There is a related discussion here: #2580).
86830fd to
84c560e
Compare
ryanking13
left a comment
There was a problem hiding this comment.
Thanks @bitsondatadev! I have some minor comment otherwise looks good to me.
88ca16a to
1af5565
Compare
Thanks @ryanking13, just pushed these changes and I will update to better testing for pyodide once #2933 has a resolution. We currently depend on Arrow for local writes. This looks like it's coming along well though so shouldn't be much longer. |
|
@ryanking13 let me know if there are any other changes needed. Thank you so much for all of the time and explanation you offered! |
| 'MIT License Copyright (c) 2011-2023 Hajime Senuma Permission is hereby | ||
| granted, free of charge, to any person obtaining a copy of this software and associated | ||
| documentation files (the "Software"), to deal in the Software without restriction, | ||
| including without limitation the rights to use, copy, modify, merge, publish, | ||
| distribute, sublicense, and/or sell copies of the Software, and to permit persons | ||
| to whom the Software is furnished to do so, subject to the following conditions: The | ||
| above copyright notice and this permission notice shall be included in all copies | ||
| or substantial portions of the Software. THE SOFTWARE IS PROVIDED "AS IS", WITHOUT | ||
| WARRANTY OF ANY KIND, EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES | ||
| OF MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO | ||
| EVENT SHALL THE AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES | ||
| OR OTHER LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING | ||
| FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN | ||
| THE SOFTWARE.' |
There was a problem hiding this comment.
If this is the normal text of the MIT license, could you shorten it to just license: MIT?
There was a problem hiding this comment.
lol of course :) this is what came down with the skeleton command and I think all it adds is the author's name, but MIT should be just fine.
1af5565 to
1070f48
Compare
|
Done @hoodmane! |
Oops added another package I was playing with locally |
d0d4e04 to
d419539
Compare
|
Thanks @bitsondatadev! |
|
Awesome! Thanks @ryanking13 and @hoodmane! Now we just need to get PyArrow changes in! |
Description
This adds the meta.yaml files to support pyIceberg, a Python client for the SQL table storage format for large datasets with periodic dimensional changes. This also includes the meta.yaml files for the mmh3 package.
Related: #4632
Fixes: #4644
Related discussion on setup: #4632
I will be adding tests/docs/etc.. shortly, just wanted to get the PR up for visibility.
Checklists