Skip to content

Commit 0636d8d

Browse files
a10ya10y
authored
feat[python]: support full read/write from object storage (#6022)
This adds the ability to write in PyVortex directly to object storage, along with more ergonomic configuration of object store credentials for the major PyVortex read/write paths: `vx.open`, `vx.io.read`, and `vx.io.write`, as well as `VortexWriteOptions.write`. This is using the https://docs.rs/pyo3-object_store/latest/pyo3_object_store/ crate, which is what runs the [obstore](https://developmentseed.org/obstore/latest/) Python library. Notably, we can't reuse `obstore` and all of its types directly. This is b/c of the way it works, you're required to actually link in pyo3-object_store and then expose it as a new module **within your Python bundle**. There are some other examples of this pattern: * [nutpie](https://github.com/pymc-devs/nutpie/blob/adb83e24235773d7ff97f86e1858ee9d31db0936/src/wrapper.rs#L1624-L1625) * [geoarrow](https://github.com/geoarrow/geoarrow-rs/blob/main/python/geoarrow-io/src/lib.rs#L46-L48) Sadly there's no nicely distributed types package, so I need to copy all of the pyi files here with their original MIT license. I've added a Python unit test to demonstrate using the new object store builders with simple local file system object store Resolves #5673 --------- Signed-off-by: Andrew Duffy <[email protected]>
1 parent ed75f70 commit 0636d8d

33 files changed

Lines changed: 2350 additions & 74 deletions

‎Cargo.lock‎

Lines changed: 39 additions & 0 deletions
Some generated files are not rendered by default. Learn more about customizing how changed files appear on GitHub.

‎docs/api/python/index.rst‎

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -10,5 +10,6 @@ Python API
1010
expr
1111
compress
1212
io
13+
store
1314
dataset
1415
type_aliases

‎docs/api/python/io.rst‎

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -28,3 +28,4 @@ HTTP, S3, Google Cloud Storage, and Azure Blob Storage.
2828
.. automodule:: vortex.io
2929
:members:
3030
:imported-members:
31+

‎docs/api/python/store.rst‎

Lines changed: 17 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,17 @@
1+
Object Store support
2+
====================
3+
4+
Vortex arrays support reading and writing to object storage systems such as, S3, Google Cloud Storage, and
5+
Azure Blob Storage.
6+
7+
.. autosummary::
8+
:nosignatures:
9+
10+
.. raw:: html
11+
12+
<hr>
13+
14+
.. automodule:: vortex.store
15+
:members:
16+
:imported-members:
17+

‎pyproject.toml‎

Lines changed: 3 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -68,3 +68,6 @@ known-first-party = [
6868
log_cli = true
6969
log_cli_level = "INFO"
7070
xfail_strict = true
71+
72+
[tool.basedpyright]
73+
exclude = ["vortex-python/python/vortex/_lib/store/**.pyi"]

‎vortex-io/src/write.rs‎

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -62,7 +62,7 @@ where
6262
}
6363

6464
fn shutdown(&mut self) -> impl Future<Output = io::Result<()>> {
65-
ready(Ok(()))
65+
ready(Write::flush(self))
6666
}
6767
}
6868

‎vortex-python/Cargo.toml‎

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -46,6 +46,7 @@ parking_lot = { workspace = true }
4646
pyo3 = { workspace = true, features = ["abi3", "abi3-py311"] }
4747
pyo3-bytes = { workspace = true }
4848
pyo3-log = { workspace = true }
49+
pyo3-object_store = { version = "0.7" }
4950
tokio = { workspace = true, features = ["fs", "rt-multi-thread"] }
5051
# This feature makes the underlying tracing logs to be emitted as `log` events
5152
tracing = { workspace = true, features = ["std", "log"] }

‎vortex-python/pyproject.toml‎

Lines changed: 3 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -64,6 +64,9 @@ include = [
6464
{ path = "python/vortex/py.typed", format = "sdist" },
6565
]
6666

67+
[tool.basedpyright]
68+
exclude = ["python/vortex/_lib/store/**.pyi"]
69+
6770
[dependency-groups]
6871
dev = [
6972
"basedpyright>=1.31",

‎vortex-python/python/vortex/__init__.py‎

Lines changed: 4 additions & 3 deletions
Original file line numberDiff line numberDiff line change
@@ -1,7 +1,7 @@
11
# SPDX-License-Identifier: Apache-2.0
22
# SPDX-FileCopyrightText: Copyright the Vortex contributors
33

4-
from importlib import metadata as _metadata
4+
import importlib.metadata
55

66
from . import _lib, arrays, dataset, expr, file, io, ray, registry, scan
77
from ._lib.arrays import ( # pyright: ignore[reportMissingModuleSource]
@@ -84,11 +84,12 @@
8484
assert _lib, "Ensure we eagerly import the Vortex native library"
8585

8686
# Resolve the installed distribution version so it is available as vortex.__version__.
87+
8788
__version__ = "unknown"
8889
try:
8990
# Try to read the installed distribution version for the Python package name.
90-
__version__ = _metadata.version("vortex-data")
91-
except _metadata.PackageNotFoundError:
91+
__version__ = importlib.metadata.version("vortex-data")
92+
except importlib.metadata.PackageNotFoundError:
9293
# If the distribution is not installed, keep the unknown fallback.
9394
pass
9495

‎vortex-python/python/vortex/_lib/__init__.pyi‎

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -1,2 +1,3 @@
11
# SPDX-License-Identifier: Apache-2.0
22
# SPDX-FileCopyrightText: Copyright the Vortex contributors
3+

0 commit comments

Comments
 (0)