Updated Python packaging + CI #146

Parskatt · 2025-07-28T15:43:27Z

Introduction

This PR introduces CI for publishing wheels to pypi (I've refrained from autopushing to pypi as it's immutable, so if you screw up you have to bump package version), and additionally comes with some bugfixes to the python bindings, and automatic python stub generation (courtesy of pycolmap). The CI is currently running over at my fork, so any updates to this PR can be tested there first, before merging into master here.

Changes

Addition of cross-platform wheels for pypi through cibuildwheel in .github/workflows/build-pypi.yaml.
No longer build python package in .github/workflows/main.yaml, as python version now has some dependencies, keep main.yaml dep free.
Unify target_compile_options across cmakelists as

if(MSVC)
  target_compile_options(benchmark PRIVATE /bigobj /fp:fast)
else()
  if (MARCH_NATIVE)
    target_compile_options(benchmark PRIVATE
      -march=native -Wall -Werror -fPIC -Wno-sign-compare
      -Wno-unused-variable -ffast-math -Wno-ignored-optimization-argument)
  else()
    target_compile_options(benchmark PRIVATE
      -Wall -Werror -fPIC -Wno-sign-compare
      -Wno-unused-variable -ffast-math)
  endif()
  if (CMAKE_CXX_COMPILER_ID STREQUAL "GNU")
    target_compile_options(benchmark PRIVATE
      -Wno-maybe-uninitialized)
  endif()
endif()

Some explicit Eigen::Vector2d::Zero() to satisfy picky compilers
Follow pycolmap and use _core for the python module, and move it into a folder poselib package folder, instead of just the raw .so.
Add pyproject.toml to specify build-dependencies of the python package
Add automatic stub generation, following pycolmap
Added constructor bindings for poselib::Camera (and also fixed segfault bug).

TODO / Possible improvements

Make equivalent of current benchmark for python module to be able to test for regressions etc. We might also consider running entire posebench
Move away from dict-type wrappers for pybind and just bind the underlying classes/structs (e.g. RansacOptions etc). Currently it's a headache for me to remember the names, also easy to make mistakes.
More explicit errors. Running something like relative pose estimation with a broken camera should result in an error, currently just returns 0 inliers.
Update poselib.Camera (Camera pybind improvements #99)

Related PRs/Issues:

#72
#73
#99

no upload to pypi, scary

vlarsson · 2025-07-31T09:10:36Z

@vlarsson @pablospe Some improvements:

Simplified the stubgen (some handfixes that pycolmap needed seems not required in our case), and now builds on all arches through python instead of bash

I've added some integration with posebench, it currently runs posebench --subsample 100, which downloads the posebench data (which I temporarily put here: https://github.com/Parskatt/storage/releases/download/posebench-v0.0.1/data.zip). As part of this I also made some refactors and packaging of posebench here: https://github.com/Parskatt/posebench

Great! So currently I guess it only checks if posebench crashes or not?

TODO

We should first merge in the changes to posebench (basically just a refactor), and also IMO move over posebench to this org (see Moving this repo to Poselib org? vlarsson/posebench#4).

I agree. I can see if I can move it.

Currently the data for posebench is downloaded for each python and each arch in the CI. Probably there are ways to cache the data, or do something to prevent this repeated downloading. Should probably look into that.

Potentially we could also set it up to only run on a subset of datasets as well. So it does not have to download everything at once.

Parskatt · 2025-07-31T09:17:52Z

Yeah, I can make a subset dataset which should be fine to dl many times.

vlarsson · 2025-07-31T09:31:36Z

Regarding

Move away from dict-type wrappers for pybind and just bind the underlying classes/structs (e.g. RansacOptions etc). Currently it's a headache for me to remember the names, also easy to make mistakes.
More explicit errors. Running something like relative pose estimation with a broken camera should result in an error, currently just returns 0 inliers.

I agree, but would hold off on this until dev is merged. In dev we have split the options into AbsolutePoseOptions etc. Ideally we have the option to still pass dicts into the pybinds, but they are automatically wrapped to corresponding structs with error messages if there are extra/invalid fields. In particular for cameras, which are often stored/serialized, it is nicer to represent them as dicts instead of these pybind classes. So I would want to keep this functionality.

pablospe · 2025-07-31T11:36:00Z

pyproject.toml

+
+[project.optional-dependencies]
+test = [
+    "posebench @ git+https://github.com/Parskatt/posebench.git; python_version < \"3.14\"",


As a reminder to change in the future: https://github.com/PoseLib/posebench

pablospe · 2025-07-31T11:36:57Z

@Parskatt great, progress! Press the button "Ready for review" when you think this is ready and we go over; it looks already quite good.

Parskatt · 2025-07-31T12:04:21Z

Thanks! I need to sort out a few things with python 3.14, but then should be ready. Don't want to feature-creep this PR too much.

Also subsample, otherwise too slow.

Parskatt · 2025-08-01T03:32:10Z

(Hopefully) final update:

test command via pytest + posebench on a subset of the data that is not very heavy to run.

There are now a few new files.
uv.lock (dependency locking), pyproject.toml, the pyposelib folder, and the tests folder.

uv.lock could arguably be removed, and added to .gitignore. uv maintainers however recommend that it not be ignored.
pyproject.toml should definitely remain.
pyposelib needs to be somewhere for the package files. The naming can be arbitrary and we could move it later to a subfolder.
tests could possibly be renamed or be a subfolder of something else, it is however a pretty standard place for python packages to store tests.

I think these additions are fine, and pretty easy to revert later if we want to change the project structure.

pablospe · 2025-08-01T20:43:02Z

yes, we should keep uv.lock. What I don't see it is where did you use it? I was expecting to find something like uv pip sync. Perhaps we need to add some documentation?

vlarsson · 2025-08-04T08:26:32Z

(Hopefully) final update:

test command via pytest + posebench on a subset of the data that is not very heavy to run.

There are now a few new files. uv.lock (dependency locking), pyproject.toml, the pyposelib folder, and the tests folder.

uv.lock could arguably be removed, and added to .gitignore. uv maintainers however recommend that it not be ignored. pyproject.toml should definitely remain. pyposelib needs to be somewhere for the package files. The naming can be arbitrary and we could move it later to a subfolder. tests could possibly be renamed or be a subfolder of something else, it is however a pretty standard place for python packages to store tests.

I think these additions are fine, and pretty easy to revert later if we want to change the project structure.

Is this ready to merge now? Or do you have more changes planned?

Parskatt · 2025-08-05T04:51:45Z

@vlarsson should be ready to merge.

@pablospe I used uv for building and dep management, but did not add any instructions, no. Basically just uv sync and uv add used. Should be noted that uv is still somewhat a pain to use with c-based projects as package is very often rebuilt, even for minor changes.

cmake/LibraryConfig.cmake

pyproject.toml

pablospe · 2025-08-06T20:00:14Z

should we create a new tag: 2.0.5?

Parskatt · 2025-08-07T00:22:46Z

@pablospe +1, the wheels are prebuilt, you can try them out from the wheelhouse to see if there are any obvious issues, otherwise Id say ready.

@vlarsson bump?

pablospe · 2025-08-12T09:01:44Z

@vlarsson Should we bump the version to 2.0.5?
@Parskatt Do we need to only add the new tag, right? Or manually create a release? Or manually triggering the CI? How do we proceed here?

Parskatt · 2025-08-12T09:16:02Z

CI will trigger for both release and push, so just changing the version in appropriate places will build wheels with correct tags. Then I'll push the wheels manually.

pablospe · 2025-08-12T10:07:37Z

I created an issue and assigned to copilot :P even though it was easy, it is amazing it worked out :)
#149

Parskatt and others added 30 commits September 29, 2023 10:13

add ci for building pypi wheels and fix some platform specific issues

1998832

increment version

44f55fc

Merge branch 'PoseLib:master' into pypi-packaging

cde2635

wip

0d89a16

wip

47abf43

Update build-pypi.yaml

c2edea1

no upload to pypi, scary

rebuild macos py39 without march native

f9770ee

bench

79fa838

wip

1214d27

test posebench

2246583

test

4a33ba5

tiny test

1162f89

build on all

abe6b4f

Update build-pypi.yaml

d32e4e3

Update setup.py

58c1933

Update setup.py

7c622d2

Update build-pypi.yaml

de54a98

Update build-pypi.yaml

4715334

Update build-pypi.yaml

d95b523

Update build-pypi.yaml

321bf16

Update build-pypi.yaml

a0354fb

Update build-pypi.yaml

4905bd5

Update build-pypi.yaml

14b9389

Merge branch 'PoseLib:master' into master

6f13133

Update build-pypi.yaml

8cbcb15

Update build-pypi.yaml

d84db3b

Update setup.py

f466af9

Update problem_generator.cc

7a5091e

Update build-pypi.yaml

5835b2f

Update build-pypi.yaml

de1df6c

pablospe reviewed Jul 31, 2025

View reviewed changes

Parskatt added 6 commits July 31, 2025 21:50

there needs to be a whitespace before the ; ...

2ffbcce

use posebench --subset flag

2d40d08

Update pyproject.toml

d24568c

Also subsample, otherwise too slow.

use posebench python api instead of cli

6562d97

fix strings

cfd340d

pytest

1d7ed85

Parskatt marked this pull request as ready for review August 1, 2025 03:09

vlarsson reviewed Aug 5, 2025

View reviewed changes

cmake/LibraryConfig.cmake Show resolved Hide resolved

pyproject.toml Outdated Show resolved Hide resolved

update posebench to official source

428484e

pablospe approved these changes Aug 5, 2025

View reviewed changes

Parskatt merged commit 7e9f5f5 into PoseLib:master Aug 6, 2025
1 check passed

pablospe mentioned this pull request Aug 12, 2025

Bump version to 2.0.5 #148

Closed

Copilot AI mentioned this pull request Aug 12, 2025

Update PoseLib version to 2.0.5 with comprehensive changelog #149

Merged

Parskatt mentioned this pull request Aug 12, 2025

Create wheels.yml #95

Closed

BrewTestBot mentioned this pull request Aug 12, 2025

poselib 2.0.5 Homebrew/homebrew-core#233281

Merged

Updated Python packaging + CI #146

Updated Python packaging + CI #146

Uh oh!

Conversation

Parskatt commented Jul 28, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Introduction

Changes

TODO / Possible improvements

Related PRs/Issues:

Uh oh!

vlarsson commented Jul 31, 2025

TODO

Uh oh!

Parskatt commented Jul 31, 2025

Uh oh!

vlarsson commented Jul 31, 2025

Uh oh!

pablospe Jul 31, 2025

Choose a reason for hiding this comment

Uh oh!

pablospe commented Jul 31, 2025

Uh oh!

Parskatt commented Jul 31, 2025

Uh oh!

Parskatt commented Aug 1, 2025

Uh oh!

pablospe commented Aug 1, 2025

Uh oh!

vlarsson commented Aug 4, 2025

Uh oh!

Parskatt commented Aug 5, 2025

Uh oh!

Uh oh!

Uh oh!

Uh oh!

pablospe commented Aug 6, 2025

Uh oh!

Parskatt commented Aug 7, 2025

Uh oh!

pablospe commented Aug 12, 2025

Uh oh!

Parskatt commented Aug 12, 2025

Uh oh!

pablospe commented Aug 12, 2025

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

Parskatt commented Jul 28, 2025 •

edited

Loading