Consider per-page settings when iterating paginated properties#3377
Consider per-page settings when iterating paginated properties#3377
Conversation
34e9af5 to
3bf2c7e
Compare
3bf2c7e to
a64138e
Compare
tests/GithubObject.py
Outdated
| # commits is a PaginatedList, which should respect per_page | ||
| commits = list(reversed(comparison.commits)) | ||
| commit = commits[-1] | ||
| # files is a PaginatedList, which should respect per_page |
There was a problem hiding this comment.
This is not true. Even for explicitly paginated requests (page=1&per_page=N), the url attribute is returned without pagination information, thus overwriting url used in construction of object(s).
There was a problem hiding this comment.
So maybe we need to change all _useAttributes methods to not overwrite _url if it's set?
|
Before PyGithub/github/PaginatedList.py Line 217 in 9ca6b4b |
9ca6b4b to
2bcfcb1
Compare
|
Unit Tests are green, yay! |
|
@EnricoMi a kind reminder of this PR. We would really like to get this merged and not maintain patches. |
|
@iarspider I have to go over this once more as this is a significant change. But it should be done properly. I hope the notion of paginated properties and when to provide a |
4fdf989 to
fb367de
Compare
5f7df43 to
8a7e258
Compare
8a7e258 to
cdbb128
Compare
|
@iarspider this is finally complete, please have a look and test with your use-case. |
Thanks! I'll try to check this ASAP. |
|
@iarspider have you had a chance to take a look / test on your side? |
|
@iarspider thanks for raising this issue! |
|
@EnricoMi thanks! Sorry, I couldn't test it - I am no longer working on that project. |
This PR contains the following updates: | Package | Change | [Age](https://docs.renovatebot.com/merge-confidence/) | [Confidence](https://docs.renovatebot.com/merge-confidence/) | |---|---|---|---| | [PyGithub](https://redirect.github.com/pygithub/pygithub) | `==2.8.1` → `==2.9.0` |  |  | | [pygithub](https://redirect.github.com/pygithub/pygithub) | `==2.8.1` → `==2.9.0` |  |  | --- ### Release Notes <details> <summary>pygithub/pygithub (PyGithub)</summary> ### [`v2.9.0`](https://redirect.github.com/PyGithub/PyGithub/releases/tag/v2.9.0) [Compare Source](https://redirect.github.com/pygithub/pygithub/compare/v2.8.1...v2.9.0) ##### Notable changes ##### Lazy PyGithub objects The notion of lazy objects has been added to some PyGithub classes in version 2.6.0. This release now makes all `CompletableGithubObject`s optionally lazy (if useful). See [PyGithub/PyGithub#3403](https://redirect.github.com/PyGithub/PyGithub/pull/3403) for a complete list. In lazy mode, getting a PyGithub object does not send a request to the GitHub API. Only accessing methods and properties sends the necessary requests to the GitHub API: ```python # Use lazy mode g = Github(auth=auth, lazy=True) # these method calls do not send requests to the GitHub API user = g.get_user("PyGithub") # get the user repo = user.get_repo("PyGithub") # get the user's repo pull = repo.get_pull(3403) # get a known pull request issue = pull.as_issue() # turn the pull request into an issue # these method and property calls send requests to Github API issue.create_reaction("rocket") # create a reaction created = repo.created_at # get property of lazy object repo # once a lazy object has been fetched, all properties are available (no more requests) licence = repo.license ``` All PyGithub classes that implement `CompletableGithubObject` support lazy mode (if useful). This is only useful for classes that have methods creating, changing, or getting objects. By default, PyGithub objects are not lazy. ##### PyGithub objects with a paginated property The GitHub API has the "feature" of paginated properties. Some objects returned by the API have a property that allows for pagination. Fetching subsequent pages of that property means fetching the entire object (with all other properties) and the specified page of the paginated property. Iterating over the paginated property means fetching all other properties multiple times. Fortunately, the allowed size of each page (`per_page` is usually 300, in contrast to the "usual" `per_page` maximum of 100). Objects with paginated properties: - Commit.files - Comparison.commits - EnterpriseConsumedLicenses.users This PR makes iterating those paginated properties use the configured `per_page` setting. It further allows to specify an individual `per_page` when either retrieving such objects, or fetching paginated properties. See [Classes with paginated properties](https://pygithub.readthedocs.io/en/stable/utilities.html#utilities-classes-with-paginated-properties) for details. ##### Drop Python 3.8 support due to End-of-Life Python 3.8 reached its end-of-life September 6, 2024. Support has been removed with this release. ##### Deprecations - Method `delete` of `Reaction` is deprecated, use `IssueComment.delete_reaction`, `PullRequestComment.delete_reaction`, `CommitComment.delete_reaction` or `Issue.delete_reaction` instead. - Method `Issue.assignee` and parameter `Issue.edit(assignee=…)` are deprecated, use `Issue.assignees` and `Issue.edit(assignees=…)` instead. - Method `Organization.edit_hook` is deprecated, use `Organization.get_hook(id).edit(…)` instead. If you need to avoid `Organization.get_hook(id)` to fetch the `Hook` object from Github API, use a lazy Github instance: ```python Github(…, lazy=True).get_organization(…).get_hook(id).edit(…) ``` - Methods `Team.add_to_members` and `Team.remove_from_members` are deprecated, use `Team.add_membership` or `Team.remove_membership` instead. ##### New Features - Consider per-page settings when iterating paginated properties by [@​EnricoMi](https://redirect.github.com/EnricoMi) in [PyGithub/PyGithub#3377](https://redirect.github.com/PyGithub/PyGithub/pull/3377) - Add Secret Scanning Alerts and Improve Code Scan Alerts by [@​matt-davis27](https://redirect.github.com/matt-davis27) in [PyGithub/PyGithub#3307](https://redirect.github.com/PyGithub/PyGithub/pull/3307) ##### Improvements - More lazy objects by [@​EnricoMi](https://redirect.github.com/EnricoMi) in [PyGithub/PyGithub#3403](https://redirect.github.com/PyGithub/PyGithub/pull/3403) - Allow for enterprise base url prefixed with `api.` by [@​EnricoMi](https://redirect.github.com/EnricoMi) in [PyGithub/PyGithub#3419](https://redirect.github.com/PyGithub/PyGithub/pull/3419) - Add `throw` option to `Workflow.create_dispatch` to raise exceptions by [@​dblanchette](https://redirect.github.com/dblanchette) in [PyGithub/PyGithub#2966](https://redirect.github.com/PyGithub/PyGithub/pull/2966) - Use `GET` url or `_links.self` as object url by [@​EnricoMi](https://redirect.github.com/EnricoMi) in [PyGithub/PyGithub#3421](https://redirect.github.com/PyGithub/PyGithub/pull/3421) - Add support for `type` parameter to get\_issues by [@​nrysk](https://redirect.github.com/nrysk) in [PyGithub/PyGithub#3381](https://redirect.github.com/PyGithub/PyGithub/pull/3381) - Align implemented paths with OpenAPI spec by [@​EnricoMi](https://redirect.github.com/EnricoMi) in [PyGithub/PyGithub#3413](https://redirect.github.com/PyGithub/PyGithub/pull/3413) - Add suggested OpenAPI schemas by [@​EnricoMi](https://redirect.github.com/EnricoMi) in [PyGithub/PyGithub#3411](https://redirect.github.com/PyGithub/PyGithub/pull/3411) - Apply OpenAPI schemas by [@​EnricoMi](https://redirect.github.com/EnricoMi) in [PyGithub/PyGithub#3412](https://redirect.github.com/PyGithub/PyGithub/pull/3412) ##### Bug Fixes - Fix `PaginatedList.totalCount` returning 0 with GitHub deprecation notices by [@​odedperezcodes](https://redirect.github.com/odedperezcodes) in [PyGithub/PyGithub#3382](https://redirect.github.com/PyGithub/PyGithub/pull/3382) - Use default type if known type is not supported by [@​EnricoMi](https://redirect.github.com/EnricoMi) in [PyGithub/PyGithub#3365](https://redirect.github.com/PyGithub/PyGithub/pull/3365) ##### Maintenance - Deprecate `Reaction.delete` by [@​iarspider](https://redirect.github.com/iarspider) in [PyGithub/PyGithub#3435](https://redirect.github.com/PyGithub/PyGithub/pull/3435) - Deprecate `Issue.assignee` by [@​EnricoMi](https://redirect.github.com/EnricoMi) in [PyGithub/PyGithub#3366](https://redirect.github.com/PyGithub/PyGithub/pull/3366) - Deprecate `Orginization.edit_hook` by [@​EnricoMi](https://redirect.github.com/EnricoMi) in [PyGithub/PyGithub#3404](https://redirect.github.com/PyGithub/PyGithub/pull/3404) - Deprecate `Team.add_to_members` and `Team.remove_from_members` by [@​EnricoMi](https://redirect.github.com/EnricoMi) in [PyGithub/PyGithub#3368](https://redirect.github.com/PyGithub/PyGithub/pull/3368) - Various minor OpenAPI fixes by [@​EnricoMi](https://redirect.github.com/EnricoMi) in [PyGithub/PyGithub#3375](https://redirect.github.com/PyGithub/PyGithub/pull/3375) - Update test key pair by [@​EnricoMi](https://redirect.github.com/EnricoMi) in [PyGithub/PyGithub#3453](https://redirect.github.com/PyGithub/PyGithub/pull/3453) - Pin CI lint Python version to 3.13 by [@​EnricoMi](https://redirect.github.com/EnricoMi) in [PyGithub/PyGithub#3406](https://redirect.github.com/PyGithub/PyGithub/pull/3406) - Improve error message on replay data mismatch by [@​EnricoMi](https://redirect.github.com/EnricoMi) in [PyGithub/PyGithub#3385](https://redirect.github.com/PyGithub/PyGithub/pull/3385) and [PyGithub/PyGithub#3386](https://redirect.github.com/PyGithub/PyGithub/pull/3386) - Disable sleeps in tests by [@​EnricoMi](https://redirect.github.com/EnricoMi) in [PyGithub/PyGithub#3383](https://redirect.github.com/PyGithub/PyGithub/pull/3383) - Update autodoc defaults by [@​Aidan-McNay](https://redirect.github.com/Aidan-McNay) in [PyGithub/PyGithub#3369](https://redirect.github.com/PyGithub/PyGithub/pull/3369) - Add Python 3.14 to CI and tox by [@​EnricoMi](https://redirect.github.com/EnricoMi) in [PyGithub/PyGithub#3429](https://redirect.github.com/PyGithub/PyGithub/pull/3429) - Restrict PyPi release workflow permissions by [@​JLLeitschuh](https://redirect.github.com/JLLeitschuh) in [PyGithub/PyGithub#3418](https://redirect.github.com/PyGithub/PyGithub/pull/3418) - Fix OpenApi workflow by [@​EnricoMi](https://redirect.github.com/EnricoMi) in [PyGithub/PyGithub#3389](https://redirect.github.com/PyGithub/PyGithub/pull/3389) - Bump codecov/codecov-action from 3 to 5 by [@​dependabot](https://redirect.github.com/dependabot)\[bot] in [PyGithub/PyGithub#3284](https://redirect.github.com/PyGithub/PyGithub/pull/3284) - Bump actions/setup-python from 5 to 6 by [@​dependabot](https://redirect.github.com/dependabot)\[bot] in [PyGithub/PyGithub#3370](https://redirect.github.com/PyGithub/PyGithub/pull/3370) - Bump dawidd6/action-download-artifact from 3.0.0 to 3.1.4 by [@​dependabot](https://redirect.github.com/dependabot)\[bot] in [PyGithub/PyGithub#3282](https://redirect.github.com/PyGithub/PyGithub/pull/3282) - Bump github/codeql-action from 3 to 4 by [@​dependabot](https://redirect.github.com/dependabot)\[bot] in [PyGithub/PyGithub#3391](https://redirect.github.com/PyGithub/PyGithub/pull/3391) - Bump actions/upload-artifact from 4 to 5 by [@​dependabot](https://redirect.github.com/dependabot)\[bot] in [PyGithub/PyGithub#3394](https://redirect.github.com/PyGithub/PyGithub/pull/3394) - Bump actions/download-artifact from 5 to 6 by [@​dependabot](https://redirect.github.com/dependabot)\[bot] in [PyGithub/PyGithub#3393](https://redirect.github.com/PyGithub/PyGithub/pull/3393) - Drop Python 3.8 support due to EOL by [@​hugovk](https://redirect.github.com/hugovk) in [PyGithub/PyGithub#3191](https://redirect.github.com/PyGithub/PyGithub/pull/3191) - Merge changelog updates from v2.8 release branch by [@​EnricoMi](https://redirect.github.com/EnricoMi) in [PyGithub/PyGithub#3367](https://redirect.github.com/PyGithub/PyGithub/pull/3367) #### New Contributors - [@​odedperezcodes](https://redirect.github.com/odedperezcodes) made their first contribution in [PyGithub/PyGithub#3382](https://redirect.github.com/PyGithub/PyGithub/pull/3382) - [@​Aidan-McNay](https://redirect.github.com/Aidan-McNay) made their first contribution in [PyGithub/PyGithub#3369](https://redirect.github.com/PyGithub/PyGithub/pull/3369) - [@​nrysk](https://redirect.github.com/nrysk) made their first contribution in [PyGithub/PyGithub#3381](https://redirect.github.com/PyGithub/PyGithub/pull/3381) - [@​matt-davis27](https://redirect.github.com/matt-davis27) made their first contribution in [PyGithub/PyGithub#3307](https://redirect.github.com/PyGithub/PyGithub/pull/3307) **Full Changelog**: <PyGithub/PyGithub@v2.8.0...v2.9.0> </details> --- ### Configuration 📅 **Schedule**: Branch creation - At any time (no schedule defined), Automerge - At any time (no schedule defined). 🚦 **Automerge**: Disabled by config. Please merge this manually once you are satisfied. ♻ **Rebasing**: Whenever PR becomes conflicted, or you tick the rebase/retry checkbox. 🔕 **Ignore**: Close this PR and you won't be reminded about these updates again. --- - [ ] <!-- rebase-check -->If you want to rebase/retry this PR, check this box --- This PR was generated by [Mend Renovate](https://mend.io/renovate/). View the [repository job log](https://developer.mend.io/github/lettuce-financial/github-bot-signed-commit). <!--renovate-debug:eyJjcmVhdGVkSW5WZXIiOiI0My42Ni40IiwidXBkYXRlZEluVmVyIjoiNDMuNjYuNCIsInRhcmdldEJyYW5jaCI6Im1haW4iLCJsYWJlbHMiOltdfQ==-->
…#3377 patterns Co-authored-by: EnricoMi <[email protected]> Agent-Logs-Url: https://github.com/PyGithub/PyGithub/sessions/d885b909-c041-400f-9b6a-f3d87aa5118c
The GitHub API has the "feature" of paginated properties. Some objects returned by the API have a property that allows for pagination. Fetching subsequent pages of that property means fetching the entire object (with all other properties) and the specified page of the paginated property. Iterating over the paginated property means fetching all other properties multiple times. Fortunately, the allowed size of each page (`per_page` is usually 300, in contrast to the "usual" `per_page` maximum of 100). Objects with paginated properties: - `Commit.files` - `Comparison.commits` - `EnterpriseConsumedLicenses.users` This PR makes iterating those paginated properties use the configured `per_page` setting. It further allows to specify an individual `per_page` when either retrieving such objects, or fetching paginated properties. Methods returning objects with paginated properties: - `Repository.compare` - `Repository.get_commit` - `Enterprise.get_consumed_licenses` Accessing paginated properties with individual `per_page` value: - `Commit.get_files` - `Comparison.get_commits` - `EnterpriseConsumedLicenses.get_users` Note: setting the `Github(per_page=…)` to a value other than the default of 30 will cause paginated properties to use that setting, unless a different value is specified when retrieving such objects or their paginated properties via methods listed above. For instance, `Commit.files` fetches paged with that number of files. Before, 300 files were fetched with each request. --------- Co-authored-by: iarspider <[email protected]> Co-authored-by: EnricoMi <[email protected]>
The GitHub API has the "feature" of paginated properties. Some objects returned by the API have a property that allows for pagination. Fetching subsequent pages of that property means fetching the entire object (with all other properties) and the specified page of the paginated property. Iterating over the paginated property means fetching all other properties multiple times. Fortunately, the allowed size of each page (
per_pageis usually 300, in contrast to the "usual"per_pagemaximum of 100).Objects with paginated properties:
Commit.filesComparison.commitsEnterpriseConsumedLicenses.usersThis PR makes iterating those paginated properties use the configured
per_pagesetting.It further allows to specify an individual
per_pagewhen either retrieving such objects, or fetching paginated properties.Methods returning objects with paginated properties:
Repository.compareRepository.get_commitEnterprise.get_consumed_licensesAccessing paginated properties with individual
per_pagevalue:Commit.get_filesComparison.get_commitsEnterpriseConsumedLicenses.get_usersNote: setting the
Github(per_page=…)to a value other than the default of 30 will cause paginated properties to use that setting, unless a different value is specified when retrieving such objects or their paginated properties via methods listed above. For instance,Commit.filesfetches paged with that number of files. Before, 300 files were fetched with each request.