Skip to content

Commit 341c2ad

Browse files
nchammascloud-fan
authored andcommitted
[SPARK-60879][INFRA] Speed up build status update workflow
### What changes were proposed in this pull request? Following on to #59829, refactor the scheduled build status updater (`update_build_status.yml`) completely: - Use batched GraphQL requests instead of per-PR REST requests. - Only update PR checks that need to be changed. Do a full sync periodically to catch build updates not associated with PR updates (e.g. a job is rerun with no new commit). - Clean up the log to only show key metrics and updates made. - Move the core logic into its own JavaScript file next to the workflow. Example of cleaned up log: ```text #60090 1dac863: queued -> in_progress #59954 48a5471: in_progress -> success 726 PRs checked, 36 skipped. Build checks: 7 updated, 2 created, 681 unchanged. ``` ### Why are the changes needed? Currently, an update pass takes about 10 minutes on a 15-minute schedule. That slows down status updates and makes changes to the workflow painful to test. Nearly all of that time goes to per-PR REST reads for PRs that haven't changed in weeks, and to rewriting checks that haven't changed. Fork runs usually finish within a few hours of a push. Of the latest fork runs on 690 open PRs, 8 were rerun more than 3 days after the push. The full sync picks those up. | | Before | After | |---|---|---| | Time per pass | ~10 min | ~45-60 s (~3-5 min for a full sync) | | PRs synced per pass | ~725 (all open) | ~170 | | API reads per pass | ~1,480 REST | 7 GraphQL | | Check updates per pass | ~685 | ~3 | | Log lines per pass | ~3,400 | ~10-15 | | Runner time per day | ~16 h | ~3 h | ### Does this PR introduce _any_ user-facing change? No, not for end-users. For contributors: `Build` checks update the same way, just sooner. A fork run rerun more than 3 days after the PR was last updated shows up on the next full sync, which is scheduled every 4 hours, or on the next pass after any PR activity (e.g. a comment). Contributor guidance now explains enabling Actions, using a non-master branch, and retriggering builds. To keep fork master as an upstream-sync branch, new Build checks for PRs from master provide `action_required` guidance rather than linking to a build, even if an older workflow ran there; existing run-linked checks continue updating. ### How was this patch tested? Via an agent: I replayed the old and new scripts read-only against live `apache/spark` PRs, intercepting writes. The new script made no write the old one didn't. All but one of the updates it skipped were no-ops. The remaining one was a stuck rerun on a PR outside the 3-day window, which the full sync covers. A full-sync replay synced all 727 open PRs. I also replayed both scripts with existing checks hidden, so every check had to be created. 724 of 725 creates matched, and each new check had its fork run's state. The one difference was a PR pushed a few minutes earlier with no fork run yet, which the new script left for the next pass. I ran quick and full syncs locally through `actions/github-script` against live PR data, intercepting writes and simulating responses, including ignored rerun updates. They checked 169 and 729 PRs in 57 seconds and 264 seconds, and logged 7 and 8 lines. All 26 focused tests passed, covering unchanged checks, applied and ignored updates, new checks, PRs from a fork's `master`, and errors. The test and dry-run harnesses load the JavaScript module directly. Replaying the 3 open PRs from a fork's `master` with their checks hidden, each got `action_required`. Tested on GitHub against my personal fork with a real `GITHUB_TOKEN`: a full read-only sync checked 728 PRs in 4m30s using 18 GraphQL points. A temporary fork PR verified check creation, updating the same check from queued to cancelled, and zero API writes on an unchanged repeat, with no duplicate checks. ### Was this patch authored or co-authored using generative AI tooling? Generated-by: GitHub Copilot Closes #60098 from nchammas/build-status-optimization. Authored-by: Nicholas Chammas <[email protected]> Signed-off-by: Wenchen Fan <[email protected]> skip-checks: true
1 parent 749cf78 commit 341c2ad

2 files changed

Lines changed: 431 additions & 396 deletions

File tree

Lines changed: 394 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,394 @@
1+
/*
2+
* Licensed to the Apache Software Foundation (ASF) under one or more
3+
* contributor license agreements. See the NOTICE file distributed with
4+
* this work for additional information regarding copyright ownership.
5+
* The ASF licenses this file to You under the Apache License, Version 2.0
6+
* (the "License"); you may not use this file except in compliance with
7+
* the License. You may obtain a copy of the License at
8+
*
9+
* http://www.apache.org/licenses/LICENSE-2.0
10+
*
11+
* Unless required by applicable law or agreed to in writing, software
12+
* distributed under the License is distributed on an "AS IS" BASIS,
13+
* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
14+
* See the License for the specific language governing permissions and
15+
* limitations under the License.
16+
*/
17+
18+
'use strict';
19+
20+
module.exports = async ({github, context, core, env = process.env, log = console}) => {
21+
// Open PRs are read with one GraphQL query per page of PRs. For each PR it returns
22+
// the Build checks on the head commit (statusCheckRollup.commit) and the fork's
23+
// workflow runs for that commit (headRef.target), both as GitHub Actions check
24+
// suites: fork runs are Actions runs, and Build checks written with the GITHUB_TOKEN
25+
// land in Actions suites too. On the PR side, only suites containing Build checks
26+
// are relevant; unrelated upstream workflow runs must not fill the page.
27+
//
28+
// A PR rarely has more than a few such suites. A fork's default branch can collect
29+
// thousands from scheduled workflows, but suites are listed oldest first and a push
30+
// creates its runs' suites right away, so they are still among the first. Should a
31+
// page ever not hold everything a PR needs, the PR fails loudly rather than being
32+
// synced from partial data.
33+
//
34+
// GitHub aborts a GraphQL request after 10 seconds and allows 60 seconds of GraphQL
35+
// CPU time per minute. A page of 25 PRs takes about 3 seconds, so each page is
36+
// followed by a pause as long as it took.
37+
// https://docs.github.com/en/graphql/overview/rate-limits-and-query-limits-for-the-graphql-api
38+
const GITHUB_ACTIONS_APP_ID = 15368
39+
const PR_PAGE_SIZE = 25
40+
// Name of the mirrored PR check, not the workflow.
41+
const BUILD_CHECK_NAME = 'Build'
42+
43+
// Quick syncs cover only open PRs updated in the last QUICK_SYNC_LOOKBACK_DAYS. A
44+
// push, comment, review, or label change updates a PR, but a check change does not.
45+
// Fork runs usually finish within hours of a push, and most reruns happen within
46+
// days. A later rerun, or a gap in quick syncs longer than the lookback, is picked
47+
// up by the next full sync, which covers all open PRs. Both schedules are defined
48+
// in update_build_status.yml.
49+
const QUICK_SYNC_LOOKBACK_DAYS = 3
50+
const fullSync = env.FULL_SYNC == 'true'
51+
const updatedSince = fullSync ? 0
52+
: Date.now() - QUICK_SYNC_LOOKBACK_DAYS * 24 * 60 * 60 * 1000
53+
54+
// A run-linked Build check points at the fork's build_main.yml run for the PR branch.
55+
// Right after a push that run may not be registered yet, so a PR updated in the last
56+
// NEW_RUN_GRACE_MINUTES without a usable run is left for a later pass instead of
57+
// receiving action_required guidance.
58+
const BUILD_WORKFLOW = '.github/workflows/build_main.yml'
59+
const NEW_RUN_GRACE_MINUTES = 5
60+
const runGraceSince = Date.now() - NEW_RUN_GRACE_MINUTES * 60 * 1000
61+
62+
const prPageQuery = `
63+
query($owner: String!, $repo: String!, $size: Int!, $cursor: String, $appId: Int!) {
64+
repository(owner: $owner, name: $repo) {
65+
pullRequests(states: OPEN, first: $size, after: $cursor,
66+
orderBy: {field: UPDATED_AT, direction: DESC}) {
67+
pageInfo { hasNextPage endCursor }
68+
nodes {
69+
number
70+
updatedAt
71+
headRefOid
72+
statusCheckRollup {
73+
commit {
74+
oid
75+
checkSuites(first: 10, filterBy: {
76+
appId: $appId, checkName: "${BUILD_CHECK_NAME}"
77+
}) {
78+
pageInfo { hasNextPage }
79+
nodes {
80+
checkRuns(first: 10,
81+
filterBy: {
82+
checkName: "${BUILD_CHECK_NAME}", checkType: LATEST
83+
}) {
84+
pageInfo { hasNextPage }
85+
nodes { databaseId status conclusion text }
86+
}
87+
}
88+
}
89+
}
90+
}
91+
headRef {
92+
name
93+
repository { name owner { login } }
94+
target {
95+
oid
96+
... on Commit {
97+
checkSuites(first: 10, filterBy: {appId: $appId}) {
98+
pageInfo { hasNextPage }
99+
nodes {
100+
status
101+
conclusion
102+
branch { name }
103+
workflowRun { databaseId file { path } }
104+
}
105+
}
106+
}
107+
}
108+
}
109+
}
110+
}
111+
}
112+
}`
113+
114+
// Yield the open PRs updated since updatedSince one page at a time, most recently
115+
// updated first, pausing after each page as long as it took (see above). A PR
116+
// updated during the pass moves to the front and may be missed, but it is then
117+
// recent enough for the next pass.
118+
async function* listOpenPRs() {
119+
let cursor = null
120+
for (;;) {
121+
const started = Date.now()
122+
const data = await github.graphql(prPageQuery, {
123+
...context.repo,
124+
size: PR_PAGE_SIZE,
125+
cursor,
126+
appId: GITHUB_ACTIONS_APP_ID
127+
})
128+
const took = Date.now() - started
129+
const page = data.repository.pullRequests
130+
const recent = page.nodes.filter(node =>
131+
Date.parse(node.updatedAt) >= updatedSince)
132+
yield recent
133+
if (recent.length < page.nodes.length || !page.pageInfo.hasNextPage) {
134+
return
135+
}
136+
cursor = page.pageInfo.endCursor
137+
await new Promise(resolve => setTimeout(resolve, took))
138+
}
139+
}
140+
141+
// Flatten a PR from a page into its number, updatedAt, head sha, fork owner, repo and
142+
// branch, and:
143+
// - buildChecks: the head commit's Build checks.
144+
// - forkRuns: the fork's workflow runs for the head commit, by run ID, and
145+
// forkRunsTruncated: whether the page held only some of them.
146+
// - buildRun: the fork's newest build_main.yml run for the PR branch, if any. The
147+
// branch matters because the same commit can have runs from pushes to different
148+
// branches, including a skipped master run and a build on the PR branch.
149+
// Statuses and conclusions are lowercased to match the check-runs REST API.
150+
// Return null if the fork branch is gone or not at the PR head commit (the branch
151+
// was just pushed to and the PR has yet to follow). Throw if the page does not hold
152+
// all of the PR's Build checks.
153+
const toPR = (node) => {
154+
const target = node.headRef?.target
155+
const commit = node.statusCheckRollup?.commit
156+
if (target?.oid != node.headRefOid || (commit && commit.oid != node.headRefOid)) {
157+
return null
158+
}
159+
// A commit without any checks or statuses has no rollup.
160+
const suites = commit?.checkSuites.nodes ?? []
161+
if (commit?.checkSuites.pageInfo.hasNextPage
162+
|| suites.some(s => s.checkRuns.pageInfo.hasNextPage)) {
163+
throw new Error('The PR head commit has more check suites or Build checks '
164+
+ 'than a page holds')
165+
}
166+
// checkName filtering can repeat suites; process each Build check only once.
167+
const checks = new Map(suites.flatMap(s => s.checkRuns.nodes).map(cr => [cr.databaseId, cr]))
168+
const lower = (value) => value?.toLowerCase()
169+
const runs = target.checkSuites.nodes.filter(s => s.workflowRun).map(s => ({
170+
id: s.workflowRun.databaseId,
171+
status: lower(s.status),
172+
conclusion: lower(s.conclusion),
173+
workflow: s.workflowRun.file?.path,
174+
branch: s.branch?.name
175+
}))
176+
return {
177+
number: node.number,
178+
updatedAt: node.updatedAt,
179+
sha: node.headRefOid,
180+
forkOwner: node.headRef.repository.owner.login,
181+
forkRepo: node.headRef.repository.name,
182+
branch: node.headRef.name,
183+
buildChecks: [...checks.values()].map(cr => ({
184+
id: cr.databaseId,
185+
status: lower(cr.status),
186+
conclusion: lower(cr.conclusion),
187+
text: cr.text
188+
})),
189+
forkRuns: new Map(runs.map(run => [String(run.id), run])),
190+
forkRunsTruncated: target.checkSuites.pageInfo.hasNextPage,
191+
buildRun: runs.findLast(run => run.workflow == BUILD_WORKFLOW
192+
&& run.branch == node.headRef.name)
193+
}
194+
}
195+
196+
// The fork run ID a Build check points at, stored as JSON in its output text, or
197+
// undefined if the text is not that JSON (e.g. a check from an older version or a
198+
// manual run).
199+
const runIdOf = (cr) => {
200+
try {
201+
return JSON.parse(cr.text).run_id
202+
} catch {
203+
return undefined
204+
}
205+
}
206+
207+
// The Build check state that shows a fork run's state. The check-runs API accepts
208+
// fewer values than a run reports: the other pending statuses (requested, waiting,
209+
// pending) become queued, and the other conclusions (startup_failure, stale, or
210+
// none), which all mean the run produced no test result, become failure.
211+
const checkState = (run) => {
212+
if (run.status != 'completed') {
213+
return {status: run.status == 'in_progress' ? 'in_progress' : 'queued'}
214+
}
215+
const conclusions = ['success', 'failure', 'neutral', 'cancelled', 'timed_out',
216+
'action_required', 'skipped']
217+
return {
218+
status: 'completed',
219+
conclusion: conclusions.includes(run.conclusion) ? run.conclusion : 'failure'
220+
}
221+
}
222+
const label = (state) => state.status == 'completed' ? state.conclusion : state.status
223+
const sameState = (a, b) => a.status == b.status
224+
&& (a.status != 'completed' || a.conclusion == b.conclusion)
225+
226+
// Record PR-specific failures and fail the step after syncing the rest.
227+
// A failed page query instead aborts the pass.
228+
const failedPRs = new Set()
229+
230+
// The log lists only the Build checks a pass creates or updates, and errors, each
231+
// with its PR and head commit. Everything else is counted for a summary at the end.
232+
const count = {prs: 0, skipped: 0, updated: 0, created: 0, unchanged: 0, notApplied: 0}
233+
const prUrl = `https://github.com/${context.repo.owner}/${context.repo.repo}/pull/`
234+
235+
log.log(fullSync ? 'Full sync: all open PRs'
236+
: 'Quick sync: open PRs updated in the last ' + QUICK_SYNC_LOOKBACK_DAYS + ' days')
237+
for await (const prs of listOpenPRs()) {
238+
for (const node of prs) {
239+
count.prs++
240+
const ref = prUrl + node.number + ' ' + node.headRefOid.slice(0, 7)
241+
let pr
242+
try {
243+
pr = toPR(node)
244+
} catch (error) {
245+
log.error(ref + ': ' + error.message + '; raise the limits in prPageQuery')
246+
failedPRs.add(node.number)
247+
continue
248+
}
249+
if (!pr) {
250+
count.skipped++
251+
continue
252+
}
253+
254+
// Sync every Build check that points at a fork run, not just the first: a commit
255+
// can carry more than one (e.g. left by the notify workflow this one replaced).
256+
// An action_required guidance check is not linked to a run.
257+
let syncableBuildCheck = false
258+
for (const cr of pr.buildChecks) {
259+
if (cr.conclusion == 'action_required') {
260+
continue
261+
}
262+
const runId = runIdOf(cr)
263+
const run = pr.forkRuns.get(String(runId))
264+
if (!run) {
265+
if (runId && pr.forkRunsTruncated) {
266+
// The run may be among the fork's check suites the page left out. Keep
267+
// the backfill from replacing the check, and fail the PR to get noticed.
268+
log.error(ref + ': fork run ' + runId + ' not in the first fork check '
269+
+ 'suites; raise the limits in prPageQuery')
270+
failedPRs.add(pr.number)
271+
syncableBuildCheck = true
272+
}
273+
// Otherwise the check's output is unusable or its fork run was deleted, so
274+
// it can never be synced; the backfill below replaces it.
275+
continue
276+
}
277+
syncableBuildCheck = true
278+
279+
// Leave a check alone if it already shows the run's state, as nearly all do.
280+
const state = checkState(run)
281+
if (sameState(cr, state)) {
282+
count.unchanged++
283+
continue
284+
}
285+
const change = label(cr) + ' -> ' + label(state)
286+
try {
287+
const {data: updated} = await github.rest.checks.update({
288+
...context.repo,
289+
check_run_id: cr.id,
290+
...state
291+
})
292+
// A successful request can leave the state unchanged, notably when trying
293+
// to reopen a completed check after a rerun.
294+
if (sameState(cr, updated)) {
295+
count.unchanged++
296+
} else {
297+
log.log(ref + ': ' + label(cr) + ' -> ' + label(updated))
298+
count.updated++
299+
}
300+
if (!sameState(updated, state)) {
301+
count.notApplied++
302+
}
303+
} catch (error) {
304+
log.error(ref + ': ' + change + ' failed: ' + error.message)
305+
failedPRs.add(pr.number)
306+
}
307+
}
308+
309+
// No syncable Build check: link one to the PR branch's build_main.yml run, or
310+
// create action_required guidance if no run exists, it was skipped, or the PR
311+
// is from fork master. Fork master is reserved for syncing upstream; new PR
312+
// builds must use other branches, even if an older workflow ran on master.
313+
// Existing run-linked checks are still synchronized above.
314+
if (!syncableBuildCheck) {
315+
const run = pr.branch == 'master' || pr.buildRun?.conclusion == 'skipped'
316+
? undefined : pr.buildRun
317+
if (!run) {
318+
// Already provided guidance, or still within the run-registration grace period.
319+
if (pr.buildChecks.some(cr => cr.conclusion == 'action_required')
320+
|| Date.parse(pr.updatedAt) > runGraceSince) {
321+
count.skipped++
322+
continue
323+
}
324+
}
325+
326+
const state = run ? checkState(run)
327+
: {status: 'completed', conclusion: 'action_required'}
328+
const check = {
329+
...context.repo, name: BUILD_CHECK_NAME, head_sha: pr.sha, ...state
330+
}
331+
const change = 'none -> ' + label(state)
332+
try {
333+
if (run) {
334+
const actions_url = 'https://github.com/' + pr.forkOwner + '/' + pr.forkRepo
335+
+ '/actions/runs/' + run.id
336+
check.output = {
337+
title: 'Test results',
338+
summary: '[See test results](' + actions_url + ')\n\n'
339+
+ 'If the tests fail for reasons unrelated to this pull request, '
340+
+ 'please rerun the workflow in your forked repository.\n'
341+
+ 'If the failures are related to this pull request, '
342+
+ 'please investigate them and push follow-up changes.',
343+
text: JSON.stringify({
344+
owner: pr.forkOwner,
345+
repo: pr.forkRepo,
346+
run_id: run.id
347+
})
348+
}
349+
check.details_url = actions_url
350+
} else {
351+
// No usable run, or a fork-master PR: explain the required setup.
352+
check.output = {
353+
title: 'PR build setup requires attention',
354+
summary: `
355+
A supported Build check is not available for this PR head commit.
356+
357+
1. If you did not enable GitHub Actions in your forked repository, please enable it by clicking the button as shown in the image below. See also [Managing GitHub Actions settings for a repository](https://docs.github.com/en/repositories/managing-your-repositorys-settings-and-features/enabling-features-for-your-repository/managing-github-actions-settings-for-a-repository) for more details.
358+
2. For PR build status, use a branch other than \`master\` in your fork.`
359+
+ (pr.branch == 'master' ? '' : `
360+
361+
After fixing the cause, push a new commit to your PR branch to start the build.`),
362+
images: [
363+
{
364+
alt: 'enabling workflows button',
365+
image_url: 'https://raw.githubusercontent.com/apache/spark/master/.github/workflows/images/workflow-enable-button.png'
366+
}
367+
]
368+
}
369+
}
370+
await github.rest.checks.create(check)
371+
log.log(ref + ': ' + change)
372+
count.created++
373+
} catch (error) {
374+
log.error(ref + ': ' + change + ' failed: ' + error.message)
375+
failedPRs.add(pr.number)
376+
}
377+
}
378+
}
379+
}
380+
381+
log.log(count.prs + ' PRs checked, ' + count.skipped + ' skipped. '
382+
+ 'Build checks: ' + count.updated + ' updated, ' + count.created + ' created, '
383+
+ count.unchanged + ' unchanged.')
384+
385+
if (count.notApplied > 0) {
386+
core.warning(count.notApplied + ' check update(s) did not reach the requested '
387+
+ 'state. GitHub cannot reopen completed checks.')
388+
}
389+
390+
if (failedPRs.size > 0) {
391+
core.setFailed('Failed to sync ' + failedPRs.size + ' PR(s), will retry next pass: #'
392+
+ [...failedPRs].join(', #'))
393+
}
394+
};

0 commit comments

Comments
 (0)