Skip to content

Expose package version as bytewax.__version__ - #637

Merged
cheeeee merged 8 commits into
bytewax:mainfrom
Shahzeb-DataEngineer:feat/expose-version-attribute
Oct 24, 2026
Merged

cheeeee merged 8 commits into
bytewax:mainfrom
Shahzeb-DataEngineer:feat/expose-version-attribute

Conversation

@Shahzeb-DataEngineer

Copy link
Copy Markdown
Contributor

Description

Adds a top-level __init__.py for the bytewax package that exposes the installed version as bytewax.__version__, read from the package metadata via importlib.metadata:

import bytewax
print(bytewax.__version__)

This makes it easy to sanity-check which Bytewax build is actually installed and to report the version when filing issues. If the package metadata is unavailable (for example, running from an unbuilt source checkout), __version__ falls back to "unknown" instead of raising.

The __init__ is intentionally lightweight and does not import any submodules, so importing the package stays cheap and does not pull in the compiled extension.

Also adds a test asserting bytewax.__version__ matches the installed distribution version, and a changelog entry.

Closes #326

Copilot AI balanced review requested due to automatic review settings October 20, 2026 14:04

Copilot AI left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Copilot was unable to review this pull request because the user who requested the review has reached their quota limit.

An organization owner can choose the repository organization as the billing source for reviews requested by members with a paid Copilot license. Learn about Copilot code review billing.

@cheeeee

cheeeee commented Oct 20, 2026

Copy link
Copy Markdown
Collaborator

Thank you for your PR.

It would make sense to me if we could add __git_revision__ and __git_dirty__ alongside __version__, so users can identify the exact build.

The values could come from:

  • __version__: Cargo.toml, through Maturin.
  • __git_revision__: git rev-parse HEAD at build time.
  • __git_dirty__: checking for uncommitted source changes at build time.

These could be stored in a generated file included in wheels and sdists, so users wouldn't need Git installed.

@Shahzeb-DataEngineer, could you make these changes, or would it be better to split them into multiple PRs?

A new build.rs runs as part of the Maturin build and generates
pysrc/bytewax/_build_info.py with the version from Cargo.toml, the
git revision from `git rev-parse HEAD`, and the dirty state from
`git status --porcelain`, all captured at build time. The generated
file is packaged into wheels and sdists, so the exact build can be
identified from Python without Git installed. bytewax/__init__.py
falls back to importlib.metadata and "unknown" values when the
generated file is absent.

Signed-off-by: Shahzeb Ali <[email protected]>
@Shahzeb-DataEngineer

Copy link
Copy Markdown
Contributor Author

Thank you for the review, and for sketching out the design. I have added __git_revision__ and __git_dirty__ alongside __version__ in this PR. I kept it in one PR since the pieces are small and belong together, but I am happy to split it up if you prefer.

How it works:

  • A new build.rs runs as part of the Maturin build and writes a generated pysrc/bytewax/_build_info.py (gitignored, not committed). The version comes from CARGO_PKG_VERSION (so, Cargo.toml, the same value Maturin uses), the revision from git rev-parse HEAD, and the dirty flag from git status --porcelain, all captured at build time.
  • The generated file is packaged into wheels and sdists, so no Git is needed at runtime. bytewax/__init__.py imports the three values from it, and falls back to importlib.metadata / "unknown" when the file is absent, e.g. in an unbuilt source checkout. The build also never fails just because Git is unavailable.
  • pytests/test_version.py now covers all three attributes, and the CHANGELOG entry is updated.

One consequence worth naming: a build made from an sdist has no .git, so it records the revision as "unknown" (the version is still exact). Happy to adjust any of this.

@cheeeee

cheeeee commented Oct 20, 2026

Copy link
Copy Markdown
Collaborator

This is starting to look more complicated than I expected :)

Thanks for exploring this. Let's track the commit SHA and dirty-tree metadata in separate issues.

For this PR, I'd suggest returning to the original importlib.metadata approach, so we can expose the installed version without loading the native extension. (Sorry for the initial confusion)

Longer term, collecting build information through Cargo and exposing it through Rust/PyO3 could be useful, but getting the build and packaging behaviour right requires more work than I initially anticipated. Let's defer that and keep this PR focused on __version__.

Thank you!

The `repo-checks` lint job runs `mypy -p bytewax` on a fresh checkout
before the Rust build ever runs, so `bytewax._build_info` (generated by
`build.rs` during the Maturin build) does not exist yet and mypy reports
`import-not-found`. Tell mypy to ignore the missing module, the same way
the config already handles `confluent_kafka` and `jsonpickle`.

Also silence a `clippy::map_unwrap_or` warning in `build.rs` so the new
build script stays lint-clean.

Signed-off-by: Shahzeb Ali <[email protected]>
@Shahzeb-DataEngineer

Copy link
Copy Markdown
Contributor Author

Fixed the lint failure from the last run: mypy -p bytewax runs in the lint job on a fresh checkout before the Rust build ever runs, so the generated bytewax._build_info module does not exist yet and mypy reported import-not-found. Added a ignore_missing_imports override for bytewax._build_info in pyproject.toml (same pattern as the existing overrides), and also cleaned up a clippy::map_unwrap_or warning in build.rs.

The build.rs approach turned out more complex than expected. This PR goes
back to the original approach: read the installed version from the
package metadata with importlib.metadata, without loading the native
extension. Tracking the commit SHA and dirty-tree metadata in separate
issues.

Signed-off-by: Shahzeb Ali <[email protected]>
Use byte-exact upstream content for pyproject.toml, .gitignore and
CHANGELOG.md so the PR diff only shows the intended changes.

Signed-off-by: Shahzeb Ali <[email protected]>
@Shahzeb-DataEngineer

Copy link
Copy Markdown
Contributor Author

Done, this PR is back to the original importlib.metadata approach: __version__ is exposed without loading the native extension. I have removed build.rs and the git revision / dirty-tree pieces; we can track those in separate issues.

@cheeeee cheeeee added the ready for review Ready for review label Oct 21, 2026
Add-on packages such as bytewax-duckdb install modules into bytewax
without an __init__.py of their own, which worked while bytewax was a
namespace package. With the new __init__.py it is a regular package,
and an add-on in another sys.path entry than bytewax, as with an
editable install or PYTHONPATH, was no longer found.

Extend __path__ with pkgutil.extend_path, as namespace packages did
before PEP 420. Of the 17 add-ons on PyPI with code, all import in
the same environment, from a --target directory on PYTHONPATH and
with an editable bytewax, as on main; without this, 11 of them were
not found in the last two cases. The one difference from main: a path
added to sys.path after import bytewax is not searched.

The new test fails without the change and passes with it and on main.
The CHANGELOG entry says that bytewax is now a regular package.
With pysrc/bytewax/__init__.py, ruff treats bytewax as first-party
and wants third-party imports in their own group (I001). Re-sort the
ten files it reports with ruff check --fix; nothing else changes.
@cheeeee

cheeeee commented Oct 24, 2026

Copy link
Copy Markdown
Collaborator

Thank you, @Shahzeb-DataEngineer, for this PR, and for following through on the back and forth, which was on me. I decided to finish it with two small commits on top of yours:

  1. bytewax was a namespace package, and add-ons such as bytewax-duckdb install modules into it without an __init__.py of their own. With a plain __init__.py, an add-on in another sys.path entry (an editable install, PYTHONPATH) was no longer found. __init__.py now extends __path__ with pkgutil.extend_path, so bytewax.__version__ works and the add-ons are still found. A new test covers it, and the CHANGELOG entry says so.
  2. With bytewax a regular package, ruff treats it as first-party, so the imports in ten files are re-sorted, with no other change. That fixes the failing repo-checks job.

You remain the author of the change. Thanks again!

Resolve the CHANGELOG conflict with bytewax#639 by keeping both entries.

Signed-off-by: Mykyta Bozhenko <[email protected]>
@cheeeee
cheeeee merged commit 9a60f2c into bytewax:main Oct 24, 2026
45 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

ready for review Ready for review

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Make version accesible as bytewax.__version__ from python code

3 participants