Skip to content

Commit b1a9b7e

Browse files
EnricoMiiarspider
andauthored
Consider per-page settings when iterating paginated properties (#3377)
The GitHub API has the "feature" of paginated properties. Some objects returned by the API have a property that allows for pagination. Fetching subsequent pages of that property means fetching the entire object (with all other properties) and the specified page of the paginated property. Iterating over the paginated property means fetching all other properties multiple times. Fortunately, the allowed size of each page (`per_page` is usually 300, in contrast to the "usual" `per_page` maximum of 100). Objects with paginated properties: - `Commit.files` - `Comparison.commits` - `EnterpriseConsumedLicenses.users` This PR makes iterating those paginated properties use the configured `per_page` setting. It further allows to specify an individual `per_page` when either retrieving such objects, or fetching paginated properties. Methods returning objects with paginated properties: - `Repository.compare` - `Repository.get_commit` - `Enterprise.get_consumed_licenses` Accessing paginated properties with individual `per_page` value: - `Commit.get_files` - `Comparison.get_commits` - `EnterpriseConsumedLicenses.get_users` Note: setting the `Github(per_page=…)` to a value other than the default of 30 will cause paginated properties to use that setting, unless a different value is specified when retrieving such objects or their paginated properties via methods listed above. For instance, `Commit.files` fetches paged with that number of files. Before, 300 files were fetched with each request. --------- Co-authored-by: iarspider <[email protected]>
1 parent 24305f6 commit b1a9b7e

62 files changed

Lines changed: 2979 additions & 216 deletions

File tree

Some content is hidden

Large Commits have some content hidden by default. Use the searchbox below for content that may be hidden.

‎doc/utilities.rst‎

Lines changed: 86 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -33,6 +33,92 @@ Pagination
3333

3434
.. autoclass:: github.PaginatedList.PaginatedList()
3535

36+
Classes with paginated properties
37+
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
38+
39+
A few classes have paginated properties. The classes and their paginated properties are:
40+
41+
- :meth:`github.Commit.Commit.files`
42+
- :meth:`github.Comparison.Comparison.commits`
43+
- :meth:`github.EnterpriseConsumedLicenses.EnterpriseConsumedLicenses.users`
44+
45+
Iterating those properties may fetch multiple pages of items. The size of these pages is controlled by the ``per_page`` setting:
46+
47+
Objects created directly
48+
""""""""""""""""""""""""
49+
50+
Objects created via the following methods fetch the first page of items, together with the object itself. Each method
51+
provides a ``…_per_page`` parameter. The first page and all subsequent pages have this size.
52+
53+
- :meth:`github.Repository.Repository.compare` with parameter ``comparison_commits_per_page``
54+
- :meth:`github.Repository.Repository.get_commit` with parameter ``commit_files_per_page``
55+
- :meth:`github.Enterprise.Enterprise.get_consumed_licenses` with parameter ``licence_users_per_page``
56+
57+
Objects created indirectly
58+
""""""""""""""""""""""""""
59+
60+
Objects returned by other methods fetches pages with page size as configured via ``Github(per_page=…)`` (unless ``Github(per_page=30)``).
61+
62+
The classes also provide a ``get`` method to fetches the paginated property with a given page size:
63+
64+
- :meth:`github.Commit.Commit.get_files` with parameter ``commit_files_per_page``
65+
- :meth:`github.Comparison.Comparison.get_commits` with parameter ``comparison_commits_per_page``
66+
- :meth:`github.EnterpriseConsumedLicenses.EnterpriseConsumedLicenses.get_users` with parameter ``licence_users_per_page``
67+
68+
PyGithub per_page setting
69+
"""""""""""""""""""""""""
70+
71+
If none of the above ``…_per_page`` parameters is set, the value given via ``Github(per_page=…)`` is used instead (unless ``Github(per_page=30)``).
72+
If no value for ``Github(per_page=…)`` is given, the GitHub API defines a default page size for each paginated property.
73+
74+
Example:
75+
76+
.. code-block:: python
77+
78+
sha = "c4ec16a18bb4401e89967a8e395103c95cca5c2f"
79+
repo_name = "PyGithub/PyGithub"
80+
81+
# page size for paginated lists returned by get_… methods
82+
g = github.Github(per_page=100)
83+
repo = g.get_repo(repo_name)
84+
85+
# page size for the Commit.files property only
86+
commit = repo.get_commit(sha, commit_files_per_page=300)
87+
88+
# files are fetched in pages of 300, not 100,
89+
# where the first page has already been retrieved via repo.get_commit
90+
for file in commit.files:
91+
pass
92+
93+
Alternatively, a ``get_*`` method exists for each paginated property, which allows to provide a
94+
``per_page`` argument. This method is usually useful if the object is retrieved indirectly.
95+
96+
Example:
97+
98+
.. code-block:: python
99+
100+
sha1 = "18eeb269686aa5ee61ee7305ffbc3f0146c0bf5c"
101+
sha2 = "c4ec16a18bb4401e89967a8e395103c95cca5c2f"
102+
repo_name = "PyGithub/PyGithub"
103+
104+
# page size for paginated lists returned by methods
105+
g = github.Github(per_page=100)
106+
repo = g.get_repo(repo_name)
107+
108+
# page size for the Compare.commits property only
109+
compare = repo.compare(sha1, sha2, comparison_commits_per_page=1)
110+
111+
# iterating over Compare.commits happens in pages of 1
112+
commit = compare.commits[0]
113+
114+
# here, files are fetched with configured page size of 100
115+
for file in commit.files:
116+
pass
117+
118+
# here, files are fetched in pages of 300
119+
for file in commit.get_files(per_page=300):
120+
pass
121+
36122
Input classes
37123
-------------
38124

‎github/Commit.py‎

Lines changed: 43 additions & 17 deletions
Original file line numberDiff line numberDiff line change
@@ -62,7 +62,7 @@
6262
import github.NamedUser
6363
import github.PaginatedList
6464
import github.Repository
65-
from github.GithubObject import Attribute, CompletableGithubObject, NotSet, Opt, is_optional
65+
from github.GithubObject import Attribute, CompletableGithubObjectWithPaginatedProperty, NotSet, Opt, is_optional
6666
from github.PaginatedList import PaginatedList
6767

6868
if TYPE_CHECKING:
@@ -80,12 +80,15 @@
8080
from github.Repository import Repository
8181

8282

83-
class Commit(CompletableGithubObject):
83+
class Commit(CompletableGithubObjectWithPaginatedProperty):
8484
"""
8585
This class represents Commits.
8686
8787
The reference can be found here
88-
https://docs.github.com/en/rest/commits/commits#get-a-commit-object
88+
https://docs.github.com/en/rest/commits/commits#get-a-commit
89+
90+
This class has a `paginated property <https://pygithub.readthedocs.io/en/stable/utilities.html#classes-with-paginated-properties>`_.
91+
For details, see :meth:`Commit.files` or :meth:`Commit.get_files`.
8992
9093
The OpenAPI schema can be found at
9194
@@ -99,6 +102,7 @@ class Commit(CompletableGithubObject):
99102
"""
100103

101104
def _initAttributes(self) -> None:
105+
super()._initAttributes()
102106
self._author: Attribute[NamedUser] = NotSet
103107
self._comments_url: Attribute[str] = NotSet
104108
self._commit: Attribute[GitCommit] = NotSet
@@ -111,7 +115,6 @@ def _initAttributes(self) -> None:
111115
self._sha: Attribute[str] = NotSet
112116
self._stats: Attribute[CommitStats] = NotSet
113117
self._text_matches: Attribute[dict[str, Any]] = NotSet
114-
self._url: Attribute[str] = NotSet
115118

116119
def __repr__(self) -> str:
117120
return self.get__repr__({"sha": self._sha.value})
@@ -140,20 +143,28 @@ def committer(self) -> NamedUser:
140143
self._completeIfNotSet(self._committer)
141144
return self._committer.value
142145

143-
# This should be a method, but this used to be a property and cannot be changed without breaking user code
144-
# TODO: remove @property on version 3
145146
@property
146147
def files(self) -> PaginatedList[File]:
148+
"""
149+
This is a `paginated property <https://pygithub.readthedocs.io/en/stable/utilities.html#classes-with-paginated-properties>`_.
150+
151+
Iterating over this paginated list may fetch multiple pages. The size of these pages can be controlled via
152+
the ``…_per_page`` parameter of :meth:`github.Repository.Repository.get_commit`,
153+
:meth:`github.Commit.Commit.get_files`, or :meth:`github.Github`.
154+
155+
If no ``per_page`` is given, the default page size is 300. The maximum is 300.
156+
157+
At most 3000 files can be retrieved.
158+
"""
147159
return PaginatedList(
148160
github.File.File,
149161
self._requester,
150162
self.url,
151-
{},
152-
headers=None,
163+
self._pagination_parameters,
153164
list_item="files",
154165
total_count_item="total_files",
155-
firstData=self.raw_data,
156-
firstHeaders=self.raw_headers,
166+
firstData=self.raw_data if self.completed else None,
167+
firstHeaders=self.raw_headers if self.completed else None,
157168
)
158169

159170
@property
@@ -191,11 +202,6 @@ def text_matches(self) -> dict[str, Any]:
191202
self._completeIfNotSet(self._text_matches)
192203
return self._text_matches.value
193204

194-
@property
195-
def url(self) -> str:
196-
self._completeIfNotSet(self._url)
197-
return self._url.value
198-
199205
def create_comment(
200206
self,
201207
body: str,
@@ -263,6 +269,27 @@ def get_comments(self) -> PaginatedList[CommitComment]:
263269
None,
264270
)
265271

272+
def get_files(self, *, commit_files_per_page: int | None = None) -> PaginatedList[File]:
273+
"""
274+
:calls: `GET /repos/{owner}/{repo}/commits/{sha} <https://docs.github.com/en/rest/reference/repos#commits>`_
275+
276+
Identical to calling :meth:`github.Commit.Commit.files`, except that this uses the given ``per_page`` value.
277+
278+
For more details, see :meth:`github.Commit.Commit.files`.
279+
280+
:param commit_files_per_page: int Number of files retrieved per page.
281+
Iterating over the files will fetch pages of this size. The default page size is 300, the maximum is 300.
282+
At most 3000 files can be retrieved.
283+
"""
284+
return PaginatedList(
285+
github.File.File,
286+
self._requester,
287+
self.url,
288+
self._pagination_parameters_with(page=1, per_page=commit_files_per_page),
289+
headers=None,
290+
list_item="files",
291+
)
292+
266293
def get_statuses(self) -> PaginatedList[CommitStatus]:
267294
"""
268295
:calls: `GET /repos/{owner}/{repo}/statuses/{sha} <https://docs.github.com/en/rest/reference/repos#statuses>`_
@@ -335,6 +362,7 @@ def get_check_suites(self, app_id: Opt[int] = NotSet, check_name: Opt[str] = Not
335362
)
336363

337364
def _useAttributes(self, attributes: dict[str, Any]) -> None:
365+
super()._useAttributes(attributes)
338366
if "author" in attributes: # pragma no branch
339367
self._author = self._makeClassAttribute(github.NamedUser.NamedUser, attributes["author"])
340368
if "comments_url" in attributes: # pragma no branch
@@ -363,8 +391,6 @@ def _useAttributes(self, attributes: dict[str, Any]) -> None:
363391
self._stats = self._makeClassAttribute(github.CommitStats.CommitStats, attributes["stats"])
364392
if "text_matches" in attributes: # pragma no branch
365393
self._text_matches = self._makeDictAttribute(attributes["text_matches"])
366-
if "url" in attributes: # pragma no branch
367-
self._url = self._makeStringAttribute(attributes["url"])
368394

369395

370396
class CommitSearchResult(Commit):

‎github/Comparison.py‎

Lines changed: 52 additions & 20 deletions
Original file line numberDiff line numberDiff line change
@@ -41,37 +41,43 @@
4141

4242
import github.Commit
4343
import github.File
44-
from github.GithubObject import Attribute, CompletableGithubObject, NotSet
44+
from github.GithubObject import Attribute, CompletableGithubObjectWithPaginatedProperty, NotSet
4545
from github.PaginatedList import PaginatedList
4646

4747
if TYPE_CHECKING:
4848
from github.Commit import Commit
4949

5050

51-
class Comparison(CompletableGithubObject):
51+
class Comparison(CompletableGithubObjectWithPaginatedProperty):
5252
"""
5353
This class represents Comparisons.
5454
55+
The reference can be found here
56+
https://docs.github.com/en/rest/commits/commits#compare-two-commits
57+
58+
This class has a `paginated property <https://pygithub.readthedocs.io/en/stable/utilities.html#classes-with-paginated-properties>`_.
59+
For details, see :meth:`Comparison.commits` or :meth:`Comparison.get_commits`.
60+
5561
The OpenAPI schema can be found at
5662
5763
- /components/schemas/commit-comparison
5864
5965
"""
6066

6167
def _initAttributes(self) -> None:
68+
super()._initAttributes()
6269
self._ahead_by: Attribute[int] = NotSet
63-
self._base_commit: Attribute[github.Commit.Commit] = NotSet
70+
self._base_commit: Attribute[Commit] = NotSet
6471
self._behind_by: Attribute[int] = NotSet
6572
self._commits: Attribute[list[Commit]] = NotSet
6673
self._diff_url: Attribute[str] = NotSet
6774
self._files: Attribute[list[github.File.File]] = NotSet
6875
self._html_url: Attribute[str] = NotSet
69-
self._merge_base_commit: Attribute[github.Commit.Commit] = NotSet
76+
self._merge_base_commit: Attribute[Commit] = NotSet
7077
self._patch_url: Attribute[str] = NotSet
7178
self._permalink_url: Attribute[str] = NotSet
7279
self._status: Attribute[str] = NotSet
7380
self._total_commits: Attribute[int] = NotSet
74-
self._url: Attribute[str] = NotSet
7581

7682
def __repr__(self) -> str:
7783
return self.get__repr__({"url": self._url.value})
@@ -82,7 +88,7 @@ def ahead_by(self) -> int:
8288
return self._ahead_by.value
8389

8490
@property
85-
def base_commit(self) -> github.Commit.Commit:
91+
def base_commit(self) -> Commit:
8692
self._completeIfNotSet(self._base_commit)
8793
return self._base_commit.value
8894

@@ -91,20 +97,28 @@ def behind_by(self) -> int:
9197
self._completeIfNotSet(self._behind_by)
9298
return self._behind_by.value
9399

94-
# This should be a method, but this used to be a property and cannot be changed without breaking user code
95-
# TODO: remove @property on version 3
96100
@property
97-
def commits(self) -> PaginatedList[github.Commit.Commit]:
101+
def commits(self) -> PaginatedList[Commit]:
102+
"""
103+
This is a `paginated property <https://pygithub.readthedocs.io/en/stable/utilities.html#classes-with-paginated-properties>`_.
104+
105+
Iterating over this paginated list may fetch multiple pages. The size of these pages can be controlled via
106+
the ``…_per_page`` parameter of :meth:`github.Repository.Repository.compare`,
107+
:meth:`github.Comparison.Comparison.get_commits`, or :meth:`github.Github`.
108+
109+
If no ``per_page`` is given, the default page size is 250. The maximum is 1000.
110+
111+
At most 10000 commits can be retrieved.
112+
"""
98113
return PaginatedList(
99114
github.Commit.Commit,
100115
self._requester,
101116
self.url,
102-
{},
103-
headers=None,
117+
self._pagination_parameters,
104118
list_item="commits",
105119
total_count_item="total_commits",
106-
firstData=self.raw_data,
107-
firstHeaders=self.raw_headers,
120+
firstData=self.raw_data if self.completed else None,
121+
firstHeaders=self.raw_headers if self.completed else None,
108122
)
109123

110124
@property
@@ -114,6 +128,9 @@ def diff_url(self) -> str:
114128

115129
@property
116130
def files(self) -> list[github.File.File]:
131+
"""
132+
Only the first 300 changed files.
133+
"""
117134
self._completeIfNotSet(self._files)
118135
return self._files.value
119136

@@ -123,7 +140,7 @@ def html_url(self) -> str:
123140
return self._html_url.value
124141

125142
@property
126-
def merge_base_commit(self) -> github.Commit.Commit:
143+
def merge_base_commit(self) -> Commit:
127144
self._completeIfNotSet(self._merge_base_commit)
128145
return self._merge_base_commit.value
129146

@@ -147,12 +164,29 @@ def total_commits(self) -> int:
147164
self._completeIfNotSet(self._total_commits)
148165
return self._total_commits.value
149166

150-
@property
151-
def url(self) -> str:
152-
self._completeIfNotSet(self._url)
153-
return self._url.value
167+
def get_commits(self, *, comparison_commits_per_page: int | None = None) -> PaginatedList[Commit]:
168+
"""
169+
:calls: `GET /repos/{owner}/{repo}/compare/{base...:head} <https://docs.github.com/en/rest/commits/commits#compare-two-commits>`_
170+
171+
Identical to calling :meth:`github.Comparison.Comparison.commits`, except that this uses the given ``per_page`` value.
172+
173+
For more details, see :meth:`github.Comparison.Comparison.commits`.
174+
175+
:param comparison_commits_per_page: int Number of commits retrieved per page.
176+
Iterating over the commits will fetch pages of this size. The default page size is 250, the maximum is 1000.
177+
At most 10000 commits can be retrieved.
178+
"""
179+
return PaginatedList(
180+
github.Commit.Commit,
181+
self._requester,
182+
self.url,
183+
self._pagination_parameters_with(page=1, per_page=comparison_commits_per_page),
184+
headers=None,
185+
list_item="commits",
186+
)
154187

155188
def _useAttributes(self, attributes: dict[str, Any]) -> None:
189+
super()._useAttributes(attributes)
156190
if "ahead_by" in attributes: # pragma no branch
157191
self._ahead_by = self._makeIntAttribute(attributes["ahead_by"])
158192
if "base_commit" in attributes: # pragma no branch
@@ -177,5 +211,3 @@ def _useAttributes(self, attributes: dict[str, Any]) -> None:
177211
self._status = self._makeStringAttribute(attributes["status"])
178212
if "total_commits" in attributes: # pragma no branch
179213
self._total_commits = self._makeIntAttribute(attributes["total_commits"])
180-
if "url" in attributes: # pragma no branch
181-
self._url = self._makeStringAttribute(attributes["url"])

0 commit comments

Comments
 (0)