Skip to content

Upgrade python package detection#3757

Merged
AyanSinhaMahapatra merged 14 commits intodevelopfrom
improve-pypi-package-detection
Jul 1, 2024
Merged

Upgrade python package detection#3757
AyanSinhaMahapatra merged 14 commits intodevelopfrom
improve-pypi-package-detection

Conversation

@AyanSinhaMahapatra
Copy link
Member

@AyanSinhaMahapatra AyanSinhaMahapatra commented Apr 30, 2024 •
edited
Loading

Fixes #3753

Tasks

  • Reviewed contribution guidelines
  • PR is descriptively titled 📑 and links the original issue above 🔗
  • Tests pass -- look for a green checkbox ✔️ a few minutes after opening your PR
    Run tests locally to check for errors.
  • Commits are in uniquely-named feature branch and has no merge conflicts 📁
  • Updated documentation pages (if applicable)
  • Updated CHANGELOG.rst (if applicable)

@AyanSinhaMahapatra
Add basic pyproject.toml parsers #3753
77ae3ed 
Reference: #3753
Signed-off-by: Ayan Sinha Mahapatra <[email protected]>
@AyanSinhaMahapatra AyanSinhaMahapatra changed the title Add basic pyproject.toml parsers #3753 Add pyproject.toml support #3753 May 1, 2024
@AyanSinhaMahapatra AyanSinhaMahapatra force-pushed the improve-pypi-package-detection branch from 77ae3ed to 4489784 Compare May 22, 2024 11:36
@AyanSinhaMahapatra AyanSinhaMahapatra force-pushed the improve-pypi-package-detection branch from 4489784 to 4e0c8dd Compare June 21, 2024 14:56
@AyanSinhaMahapatra AyanSinhaMahapatra force-pushed the improve-pypi-package-detection branch from 4e0c8dd to c8046f1 Compare June 21, 2024 15:37
@AyanSinhaMahapatra AyanSinhaMahapatra changed the title Add pyproject.toml support #3753 Upgrade python package detection Jun 25, 2024
@AyanSinhaMahapatra AyanSinhaMahapatra force-pushed the improve-pypi-package-detection branch from 12cd136 to b467954 Compare June 28, 2024 12:29
@AyanSinhaMahapatra
Add support for pip-inspect.deplock files
fc4ada9 
Add parser for pip-inspect.deplock files generated by deplock
which has all the package metadata, i.e. the resolved versions
and the dependency relationships.

Reference: aboutcode-org/scancode.io#1262
Signed-off-by: Ayan Sinha Mahapatra <[email protected]>
@AyanSinhaMahapatra AyanSinhaMahapatra force-pushed the improve-pypi-package-detection branch from b467954 to fc4ada9 Compare June 28, 2024 12:55
@AyanSinhaMahapatra AyanSinhaMahapatra force-pushed the improve-pypi-package-detection branch from cdff730 to ac530bb Compare July 1, 2024 07:50
@AyanSinhaMahapatra
Add poetry and pyproject.toml support
bc22ee9 
* Add poetry.lock support
* Add pyproject.toml dependencies support for poetry
* Add pyproject.toml dependencies support for standard python projects
* Add poetry package assembly

Reference: #3753
Reference: #2109
Signed-off-by: Ayan Sinha Mahapatra <[email protected]>
@AyanSinhaMahapatra AyanSinhaMahapatra force-pushed the improve-pypi-package-detection branch from ac530bb to bc22ee9 Compare July 1, 2024 08:11
AyanSinhaMahapatra and others added 7 commits July 1, 2024 15:30
@AyanSinhaMahapatra
Replace deprecated macos CI runners
124da3d 
Replace macos-11 runners with macos-14 runners.

Reference: https://github.com/actions/runner-images?tab=readme-ov-file#available-images
Reference: aboutcode-org/skeleton#89
Signed-off-by: Ayan Sinha Mahapatra <[email protected]>
@AyanSinhaMahapatra
Update minimum required python version to 3.8
be4e14d 
Signed-off-by: Ayan Sinha Mahapatra <[email protected]>
@keshav-space
Merge pull request #90 from nexB/update-ci-runners
5c3e935 
Replace deprecated macos CI runners
@AyanSinhaMahapatra
Update scope and runtime/optional attributes
e2e9019 
Signed-off-by: Ayan Sinha Mahapatra <[email protected]>
@AyanSinhaMahapatra
Merge remote-tracking branch 'skeleton/main' into improve-pypi-packag…
5b734f0 
…e-detection

Signed-off-by: Ayan Sinha Mahapatra <[email protected]>
@AyanSinhaMahapatra
Replace deprecated macos-11 runner images
eae105e 
Signed-off-by: Ayan Sinha Mahapatra <[email protected]>
@AyanSinhaMahapatra
Add CHANGELOG entry and other minor updates
bc49ca0 
Signed-off-by: Ayan Sinha Mahapatra <[email protected]>
keshav-space
keshav-space approved these changes Jul 1, 2024
View reviewed changes
Copy link
Member

@keshav-space keshav-space left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks @AyanSinhaMahapatra. LGTM!
For some reason macos-13 CIs are failing.

@AyanSinhaMahapatra
Remove usage of str.removesuffix()
56aa35d 
The removesuffix attribute is only available from Python 3.9+

Signed-off-by: Ayan Sinha Mahapatra <[email protected]>
@AyanSinhaMahapatra
Copy link
Member Author

AyanSinhaMahapatra commented Jul 1, 2024 •
edited
Loading

We are failing tests in two places:

  1. tests which require linux specific configuration it seems, like rpm. Maybe this is because we don't install them in macos specific cases? Like at c06705c @pombredanne need your input here. [Edit, this might actually be the heisenbug we often get, I forgot about that actually]
  2. @keshav-space you added a removesuffix() method on a string but this is only available python 3.9+ it seems. See https://dev.azure.com/nexB/scancode-toolkit/_build/results?buildId=13829&view=logs&j=3dfa68ff-d50a-53ee-7b58-12d58c76da63&t=1b7369b2-e2fc-5889-2082-a49dded9cbd4&l=7020 and https://stackoverflow.com/questions/66683630/removesuffix-returns-error-str-object-has-no-attribute-removesuffix. I'm not sure why this only fails in macos13 though. 🤔

@keshav-space
Copy link
Member

keshav-space commented Jul 1, 2024 •
edited
Loading

  1. tests which require linux specific configuration it seems, like rpm. Maybe this is because we don't install them in macos specific cases?

Maybe we should skip these tests on macOS?

2. @keshav-space you added a removesuffix() method on a string but this is only available python 3.9+ it seems. See https://dev.azure.com/nexB/scancode-toolkit/_build/results?buildId=13829&view=logs&j=3dfa68ff-d50a-53ee-7b58-12d58c76da63&t=1b7369b2-e2fc-5889-2082-a49dded9cbd4&l=7020 and https://stackoverflow.com/questions/66683630/removesuffix-returns-error-str-object-has-no-attribute-removesuffix. I'm not sure why this only fails in macos13 though. 🤔

Ah, I missed that we're still supporting 3.8 :)

@AyanSinhaMahapatra
Copy link
Member Author

AyanSinhaMahapatra commented Jul 1, 2024

Thanks @keshav-space , merging! This seems like the heisenbug, I will deal with this seperately if this persists.

@AyanSinhaMahapatra AyanSinhaMahapatra merged commit bd36c8b into develop Jul 1, 2024
@AyanSinhaMahapatra AyanSinhaMahapatra deleted the improve-pypi-package-detection branch July 1, 2024 16:01
This was referenced Jul 1, 2024
Closed
Closed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@keshav-space keshav-space keshav-space approved these changes

Assignees

@AyanSinhaMahapatra AyanSinhaMahapatra

Labels

None yet

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

SCTK: Detect pyproject.toml correctly

2 participants

@AyanSinhaMahapatra @keshav-space