Skip to content

Conversation

@lesteve
Copy link
Member

@lesteve lesteve commented Apr 8, 2024

This was seen from time to time in the no-OpenMP build e.g. in #28757 (comment), see build log and more recently in main CI build log. The error looks like this:

  [60/249] Compiling Cython source /Users/runner/work/1/s/sklearn/cluster/_hdbscan/_linkage.pyx
  FAILED: sklearn/cluster/_hdbscan/_linkage.cpython-312-darwin.so.p/sklearn/cluster/_hdbscan/_linkage.pyx.c
  cython -M --fast-fail -3 '-X language_level=3' '-X boundscheck=False' '-X wraparound=False' '-X initializedcheck=False' '-X nonecheck=False' '-X cdivision=True' '-X profile=False' --include-dir /Users/runner/work/1/s/build/cp312 /Users/runner/work/1/s/sklearn/cluster/_hdbscan/_linkage.pyx -o sklearn/cluster/_hdbscan/_linkage.cpython-312-darwin.so.p/sklearn/cluster/_hdbscan/_linkage.pyx.c

  Error compiling Cython file:
  ------------------------------------------------------------
  ...

  cimport numpy as cnp
  from libc.float cimport DBL_MAX

  import numpy as np
  from ...metrics._dist_metrics cimport DistanceMetric64
  ^
  ------------------------------------------------------------

  /Users/runner/work/1/s/sklearn/cluster/_hdbscan/_linkage.pyx:38:0: 'sklearn/metrics/_dist_metrics.pxd' not found
  [61/249] Compiling Cython source /Users/runner/work/1/s/sklearn/utils/_isfinite.pyx
  [62/249] Compiling Cython source /Users/runner/work/1/s/sklearn/metrics/cluster/_expected_mutual_info_fast.pyx
  [63/249] Compiling C++ object sklearn/utils/_vector_sentinel.cpython-312-darwin.so.p/meson-generated_sklearn_utils__vector_sentinel.pyx.cpp.o
  [64/249] Compiling C++ object sklearn/utils/_fast_dict.cpython-312-darwin.so.p/meson-generated_sklearn_utils__fast_dict.pyx.cpp.o
  ninja: build stopped: subcommand failed.
  error: subprocess-exited-with-error
  
  × Preparing editable metadata (pyproject.toml) did not run successfully.
  │ exit code: 1
  ╰─> See above for output.
  
  note: This error originates from a subprocess, and is likely not a problem with pip.
  full command: /usr/local/miniconda/envs/testvenv/bin/python /usr/local/miniconda/envs/testvenv/lib/python3.12/site-packages/pip/_vendor/pyproject_hooks/_in_process/_in_process.py prepare_metadata_for_build_editable /var/folders/24/8k48jl6d249_n_qfxwsl6xvm0000gn/T/tmpagbgst0x
  cwd: /Users/runner/work/1/s
  Preparing editable metadata (pyproject.toml): finished with status 'error'
error: metadata-generation-failed

× Encountered error while generating package metadata.
╰─> See above for output.

note: This is an issue with the package mentioned above, not pip.
hint: See above for details.

The root cause is likely that the dependency are not correctly defined in the meson.build files. This is not detected by ninja -C build/cp312 -t missingdeps maybe because this is a ninja template .pxd.tp rather than a .pxd file?

Here are the two files that need it _dist_metrics.pxd:

❯ git grep -P 'from.*metrics.+import' -- **/*.pyx **/*.pxd
sklearn/cluster/_hdbscan/_linkage.pyx:from ...metrics._dist_metrics cimport DistanceMetric64
sklearn/cluster/_hierarchical_fast.pyx:from ..metrics._dist_metrics cimport DistanceMetric64

I have not been able to reproduce locally so I don't know for sure that this fixes the problem, but looking at the generated ninja.build file, it does seem like _dist_metrics_pxd will be a dependency of the two files that need it.

@github-actions
Copy link

github-actions bot commented Apr 8, 2024

✔️ Linting Passed

All linting checks passed. Your pull request is in excellent shape! ☀️

Generated for commit: 5863dc5. Link to the linter CI: here

Copy link
Member

@jeremiedbb jeremiedbb left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Let's try and see if we ever face the issue again.

@ogrisel ogrisel added the cython label Apr 8, 2024
@thomasjpfan thomasjpfan merged commit c1d29ce into scikit-learn:main Apr 9, 2024
@lesteve lesteve deleted the meson-build-race-condition branch April 9, 2024 14:42
@lesteve
Copy link
Member Author

lesteve commented Apr 9, 2024

Thanks if you see the same issue happening again, or one of its variants, let me know!

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

4 participants