feat(bindings/python)!: Generate stubs for Operator overloads and Scheme by chitralverma · Pull Request #6729 · apache/opendal

chitralverma · 2025-10-22T19:31:41Z

Which issue does this PR close?

Closes #6686.

Rationale for this change

This uses an alternate approach to generate the overloads previously in __base.pyi.
Instead of rendering __base.pyi directly, this approach generates a checked rust code instead which then gets submitted to pyo3-stub-gen and the resultant stub services.pyi is safely generated with all other stubs at the same time.

I aim to make this the parent class of AsyncOperator and Operator so that they can get all the overloads.

What changes are included in this PR?

Redo python generator to fix fix:(bindings/python) changes required in dev/src/generate/python.j2 #6686 along with rest of the stubs
- Overload __new__ in Operator for each service
- Overload __new__ in AysncOperator for each service
- All overloads come with typed configurations and valid docstrings
  - Config descriptions (comments ) required a bit of of preprocessing and formatting, add that.
Remove old __base.pyi
Introduce services py module containing a Scheme which can also be used in the constructors of operators

Are there any user-facing changes?

No breaking changes included.
Users get better docs and typing.

chitralverma · 2025-10-22T19:33:07Z

bindings/python/python/opendal/file.pyi

        A closed file cannot be used for further I/O operations.
        """
-    def __aenter__(self) -> typing.Self: ...
+    def __aenter__(self) -> typing_extensions.Self: ...


typing.Self was not available in python 3.10 which opendal supports, so fixed this

chitralverma · 2025-10-22T19:33:26Z

bindings/python/src/file.rs

    }

-    #[gen_stub(override_return_type(type_repr="typing.Self", imports=("typing")))]
+    #[gen_stub(override_return_type(type_repr="typing_extensions.Self", imports=("typing_extensions")))]


chitralverma · 2025-10-22T19:35:05Z

bindings/python/justfile

 [group('build')]
 stub-gen: setup
    @echo "{{ BOLD }}--- Generating Python type stubs ---{{ NORMAL }}"
+    @cargo run --quiet --manifest-path=../../dev/Cargo.toml -- generate -l python


Overloads and Scheme generates as part of the stub-gen process

This made me think that can we move stub_gen to odev instead.

This made me think that can we move stub_gen to odev instead.

actually these are 2 separate processes that cannot be merged into the binding or in odev from what i understood.

pyo3-stub-gen requires.... well, pyo3 which is available in python bindings. this step is responsible for generating most of the python stubs from rust code.

the stuff in odev uses jinja templating because it has parsing logic of services code used for java as well. if odev had lib target then it could be used as a build dependency but still would be separate from pyo3 stuff.

chitralverma · 2025-10-22T19:36:38Z

dev/src/generate/new_python.j2

@@ -0,0 +1,155 @@
+// Licensed to the Apache Software Foundation (ASF) under one


~~this file is temporary. will replace python.j2 with this.~~
done

chitralverma · 2025-10-22T19:37:17Z

dev/src/generate/python.rs

+        "etcd" | "foundationdb" | "ftp" | "hdfs" | "rocksdb" | "tikv" | "sftp" | "github"
+        | "cloudflare_kv" | "monoiofs" | "dbfs" | "surrealdb" | "d1" | "opfs" | "compfs"
+        | "lakefs" | "pcloud" | "vercel_blob" => false,


will re enable some of these in another PR

chitralverma · 2025-10-22T19:38:02Z

dev/src/generate/python.rs

+pub fn format_text(text: &str, indent: usize, max_line_length: usize) -> String {
+    // Precompute indentation string
+    let indent_str = " ".repeat(indent);


was trying to avoid an ext. crate for this this task.

chitralverma · 2025-10-22T19:38:37Z

dev/src/generate/python.rs

+        ConfigType::Bool => "builtins.bool",
+        ConfigType::Duration => "typing.Any",
        ConfigType::I64
        | ConfigType::Usize
        | ConfigType::U64
        | ConfigType::U32
-        | ConfigType::U16 => "_int",
-        ConfigType::Vec => "_strings",
-        ConfigType::String => "str",
+        | ConfigType::U16 => "builtins.int",
+        ConfigType::Vec => "typing.Any",
+        ConfigType::String => "builtins.str",


should be able to cast automatically

chitralverma · 2025-10-22T21:07:15Z

bindings/python/python/opendal/__init__.py

+from opendal._opendal import (  # noqa: F403
+    capability,
+    exceptions,
+    file,
+    layers,
+    services,
+    types,
+)
 from opendal.operator import AsyncOperator, Operator  # pyright:ignore

 __version__: builtins.str

-__all__ = _opendal.__all__  # noqa: F405 # pyright:ignore
-__all__ += ["AsyncOperator", "Operator"]  # pyright:ignore
+__all__ = [
+    "capability",
+    "exceptions",
+    "file",
+    "layers",
+    "services",
+    "types",
+    "AsyncOperator",
+    "Operator",


makes things explicit

chitralverma · 2025-10-22T21:08:12Z

dev/src/generate/new_python.j2

+{% for srv in srvs %}
+submit! {
+    gen_methods_from_python! {
+        r#"
+        import builtins
+        import typing
+        import typing_extensions
+        class Operator:
+            def __new__(cls,
+                scheme: typing.Literal["{{ snake_to_kebab_case(srv) }}"],
+                /,
+                {%- if srvs[srv].config %}


Operator overloads

chitralverma · 2025-10-22T21:08:26Z

dev/src/generate/new_python.j2

+        import builtins
+        import typing
+        import typing_extensions
+        class AsyncOperator:
+            def __new__(cls,
+                scheme: typing.Literal["{{ snake_to_kebab_case(srv) }}"],
+                /,
+                {%- if srvs[srv].config %}
+                *,


AsyncOperator overloads

chitralverma · 2025-10-22T21:09:12Z

bindings/python/src/operator.rs

    ///     The new operator.
+    #[gen_stub(skip)]
    #[new]


chitralverma · 2025-10-23T07:50:47Z

@Xuanwo this is ready for your review, when ever you're free.
This effectively finishes the refactoring for #6253 and fixes #6686.

Xuanwo

Thank you for working on this, mostly LGTM, just some questions.

Xuanwo · 2025-10-23T16:49:16Z

bindings/python/python/opendal/services.pyi

+import typing
+
+@typing.final
+class Scheme(enum.Enum):


I have a goal to finally split OpenDAL into separate crates so users can easily implement and use their own services. On the Python side, I want to dynamically load them. Is there a way to enable this?

This is not a blocker: we can still merge this PR as is, but let’s keep this idea in mind as we move forward.

separate crates how? by bindings or services ?

On the Python side, I want to dynamically load them. Is there a way to enable this?

in python world, yes via dependency groups, in rust world, yes by feature gating but in rust-python world via pyo3, i've never done this before so not sure. but then i also never did stub-gen before this, so will take a shot at it when that happens.

assuming a service level split:
makes me think that if there was a way to generate language agnostic linkable code for the services, the bindings can actually be generated via some sort of a spec driven development model. Elasticsearch and aws do this but they're server based so its easier for them.

duckdb does this as well (think of opendal services as duckdb extensions - cross platform and language agnostic) but need to think more on this. is there a issue/ discussion/ rfc for this where i can contribute better ?

The idea is shown here: #5206

In short, the OpenDAL Rust core will eventually become opendal-service-s3, opendal-service-azblob, and more. I want to know if this change can benefit Python packages. Can we avoid delivering large wheels to users?

raised astral-sh/uv#16465, which if answered will help.
maybe @messense has some suggestions from maturin side.

Xuanwo · 2025-10-23T16:50:29Z

bindings/python/justfile

 [group('build')]
 stub-gen: setup
    @echo "{{ BOLD }}--- Generating Python type stubs ---{{ NORMAL }}"
+    @cargo run --quiet --manifest-path=../../dev/Cargo.toml -- generate -l python


This made me think that can we move stub_gen to odev instead.

Xuanwo · 2025-10-23T16:50:54Z

bindings/python/mkdocs.yml

          paths: [python]
          options:
-            docstring_style: google
+            docstring_style: numpy


What's this change for? You perfer numpy style?

well before all these PRs, the code had a mix of docstring conventions which i wanted to standardize as well so i while doing this (with the help of LLMs), all the generated docstrings turned out to be in numpy convention.

today i checked that mkdocs was not rendering it properly so made this change to get it working properly.
numpy style is slightly verbose but its a very well established standard as well.

chitralverma · 2025-10-23T20:28:30Z

postgresql test is stuck.

chitralverma · 2025-10-25T10:51:46Z

@Xuanwo any other comments from your side before this can be merged?

Xuanwo

#6729 (comment) may need a follow-up, but other changes LGTM, let's move!

chitralverma added 4 commits October 22, 2025 19:47

add services module with PyScheme

1b69f2f

ensure python 3.10 compatibility file.pyi

b8207af

Add new service base

769a4e2

add config to docstrings

7dae5df

chitralverma requested review from Xuanwo, Zheaoli and messense as code owners October 22, 2025 19:31

dosubot bot added the size:XXL This PR changes 1000+ lines, ignoring generated files. label Oct 22, 2025

Merge branch 'main' into gen-services

3ea5ed7

dosubot bot added the releases-note/feat The PR implements a new feature or has a title that begins with "feat" label Oct 22, 2025

chitralverma commented Oct 22, 2025

View reviewed changes

chitralverma marked this pull request as draft October 22, 2025 19:35

chitralverma commented Oct 22, 2025

View reviewed changes

chitralverma added 2 commits October 23, 2025 01:42

fix

968e491

Add overloads to operators

8f8010c

chitralverma changed the title ~~feat(bindings/python)!: Generate new ServiceBase and add Scheme using stub-gen~~ feat(bindings/python)!: Generate stubs for Operator overloads and Scheme Oct 22, 2025

chitralverma commented Oct 22, 2025

View reviewed changes

chitralverma marked this pull request as ready for review October 22, 2025 21:18

dosubot bot added size:L This PR changes 100-499 lines, ignoring generated files. and removed size:XXL This PR changes 1000+ lines, ignoring generated files. labels Oct 22, 2025

remove and replace files

cf2d114

dosubot bot removed the size:L This PR changes 100-499 lines, ignoring generated files. label Oct 22, 2025

dosubot bot added the size:XXL This PR changes 1000+ lines, ignoring generated files. label Oct 22, 2025

chitralverma added 2 commits October 23, 2025 13:01

overloads to use pyscheme directly

ababffe

overloads to use pyscheme directly

613c3a0

chitralverma added 3 commits October 23, 2025 13:49

Add Scheme to overloads

3d736a6

Add returns section to overloads

fd97c21

update doc style

2259881

chitralverma mentioned this pull request Oct 23, 2025

new feature: Improve automatic stub generation for python bindings #6253

Closed

Xuanwo reviewed Oct 23, 2025

View reviewed changes

Merge branch 'main' into gen-services

4226720

chitralverma requested a review from Xuanwo October 23, 2025 18:33

Merge branch 'main' into gen-services

7f66a28

Merge branch 'main' into gen-services

a9c2ef2

Xuanwo approved these changes Oct 27, 2025

View reviewed changes

dosubot bot added the lgtm This PR has been approved by a maintainer label Oct 27, 2025

Xuanwo merged commit 9d8ccf0 into apache:main Oct 27, 2025
63 checks passed

chitralverma mentioned this pull request Oct 31, 2025

(bindings/python)Split OpenDAL python bindings into separate packages #6748

Open

chitralverma deleted the gen-services branch December 8, 2025 12:04

		@@ -0,0 +1,155 @@
		// Licensed to the Apache Software Foundation (ASF) under one

Comments

Conversation

chitralverma commented Oct 22, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Which issue does this PR close?

Rationale for this change

What changes are included in this PR?

Are there any user-facing changes?

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

chitralverma Oct 22, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

chitralverma Oct 22, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

chitralverma Oct 22, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

chitralverma commented Oct 23, 2025

Uh oh!

Xuanwo left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

chitralverma commented Oct 23, 2025

Uh oh!

chitralverma commented Oct 25, 2025

Uh oh!

Xuanwo left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

chitralverma commented Oct 22, 2025 •

edited

Loading

chitralverma Oct 22, 2025 •

edited

Loading

chitralverma Oct 22, 2025 •

edited

Loading

chitralverma Oct 22, 2025 •

edited

Loading