Repository navigation
Commit 1efafc4
authored
* ci: setup-java's gradle cache is the same directory as the explicit one, stored twice
Both build jobs ran `cache: gradle` on actions/setup-java AND an explicit
actions/cache step on ~/.gradle/caches. Those are not complementary: setup-java's
gradle entry is pinned to `path: [join(os.homedir(), '.gradle', 'caches')]`
(setup-java/src/cache.ts), byte-for-byte the same tree. So every run saved that
directory twice under two different keys.
The in-repo comment justifying the second cache — "setup-java's `cache: gradle`
only restores the dependency caches (modules-2), NOT build-cache-1" — was simply
false, and is corrected here rather than deleted, since it is the reason the
duplicate survived review twice.
This matters because the repo is at GitHub's 10GB Actions cache cap (measured
2026-08-11: 9.94GB across 40 entries, after LRU had already evicted the excess
from a 13.49GB peak). Constant eviction is why a "cached" quiche native build
still costs ~339s on Apple and ~204s on Linux. The duplicates measured
687MB (setup-java-macOS-arm64-gradle-*) and 264MB (setup-java-Linux-x64-gradle-*)
against the 1088MB/1373MB explicit entries that already held the same bytes.
~/.gradle/wrapper was the one thing setup-java cached that the explicit step did
not, so it gets its own step rather than being folded into the existing one.
Keeping it separately keyed is deliberate and mirrors what setup-java does
(`additionalCaches: [{name: 'gradle-wrapper', ...}]`, added for
actions/setup-java#269): the build-outputs key rotates on every *.gradle.kts
edit, and folding the wrapper in would re-download the ~150MB Gradle
distribution on each such edit.
Other workflows keep `cache: gradle` — it is their only Gradle cache.
Validated: ruamel round-trip clean on all 14 workflows; actionlint 32 findings
before and 32 after, zero new (all pre-existing shellcheck info/warnings).
* ci(apple): build the quiche natives once, then fan the rest of the job out
build-apple was one serial job of 31.7-42.8 min: build five Apple quiche natives,
then ~15-19 min of Kotlin/Native tests, then two ~8 min JVM suites, then the
device-target compile guard, then link, then publish. Nothing after the natives
depended on anything else after the natives — they shared a job, not a dependency.
natives --+-- kn-tests (macos-x64+ktlint | ios-sim | tvos+watchos | device-compile)
+-- jvm-tests (jni | ffm)
+-- link -- integration-macos-arm64
+-- publish
The largest single win is not the test sharding, it is taking `publish` off the
tail: it sat behind ~30 minutes of tests purely by colocation, while `validate`
(the caller's next job) needs the published artifacts and not the test results.
This does NOT weaken the release gate. A reusable-workflow call completes only
when ALL of its jobs complete, and fails if ANY fail, so merged.yaml's
publish -> validate -> build-apple chain still waits for every test job here.
Nothing reaches Central on a red suite; the comment at merged.yaml:253 still holds.
TWO SILENT-FAILURE TRAPS, both handled in .github/actions/apple-quiche-env:
1. The tasks skip their 10-20 minute cargo builds via
`onlyIf { !markerFile.exists() || !destStatic.exists() }`, and every marker is
a DOTFILE inside lib/ (`.built-<ver>-qlog[-jvm]-<digest>`, `.jni-built-<ver>`,
`.bssl-vendored`). actions/upload-artifact excludes hidden files unless
`include-hidden-files: true`. Without it the archives arrive without their
markers and EVERY shard silently re-runs cargo — a green run that costs far
more than the serial job it replaced. The composite action hard-fails if no
markers are restored, so this cannot regress quietly.
2. buildJvmJniShim*'s guard is an mtime comparison,
`jniSourceFile.lastModified() > markerFile.lastModified()`. Checkout writes
quiche_jni.c at job start, and a downloaded marker can carry an earlier mtime,
which reads as "source edited after the shim was built" and rebuilds. The
restored markers are touched, which is the true state: these natives were built
from this commit, by the natives job, in this same run.
Consumers use actions/cache/restore (read-only); `natives` is the single writer
for the Konan/Gradle keys, so a cache miss cannot become six jobs racing to upload
the same key. The JVM-only lanes skip the ~450MB Konan restore entirely.
The composite action also asserts ORG_GRADLE_PROJECT_version is non-empty before
any Gradle call. The pin now lives in one workflow-level `env` covering all six
jobs; if that expression ever stopped resolving, build.gradle.kts would silently
fall back to a live Maven Central query per job — reintroducing exactly the #329
split, and invisibly, since each job would still produce a plausible version.
Shard balance is a first cut, not a measured optimum: every extra macOS job
re-pays checkout + JDK + cache restore + Gradle configuration (~3-4 min), so
splitting further is not free. `Run Apple K/N tests` alone swung 884s..1151s
run-to-run before any of this, so judge changes here over several runs.
Validated: ruamel round-trip clean on all 14 workflows + the composite action;
actionlint 32 findings on main, 31 now, zero new (the resolved one is the
`>> $GITHUB_ENV` step replaced by workflow-level env).
* test(quic): await the trace lines this test asserts on, instead of a fixed beat (#334)
JvmQuicTraceCaptureTests' server-capture test synchronised on `delay(150)` and
`delay(250)` and then asserted on STATE lines — but both beats were tuned and
commented for DGRAM flush. On a loaded macOS runner that is a coin flip:
"expected >=2 fully-established server-side traces, got 1", with both connections
having established and echoed successfully (every earlier assertion passed) and
only one sink having recorded its `.Established` line yet.
The lines are recorded asynchronously from a `state.collect` collector that shares
the driver's single-threaded driverContext with the driver loop itself
(QuicheDriver.kt:349), so there is no bound on how far behind the observed echo
they can be. `TraceSink.emit` is synchronous — there is no buffer to flush, which
is why waiting longer was never the right fix either.
awaitServerTraces polls the exact condition the assertions read (N distinct
datagram paths AND N sinks carrying `.Established`) under a bounded
`withTimeout(5.seconds * testTimeScale())`. The timeout is swallowed on purpose so
the original assertions still report which invariant fell short, with the full
trace snapshot, rather than a generic timeout.
The load-bearing call site is inside the per-client coroutine, before
`stream.close()`: both clients rendezvous on the same condition, so neither tears
its connection down while the other's server-side driver still owes the trace a
line. The second call, before `finally { serverJob.cancel() }`, is usually already
satisfied.
Also fixes a latent CME this exposed: `sinks.map { it.toList() }` iterated a
Collections.synchronizedList unguarded (element access is synchronized, iteration
is not). It was masked by running only after serverJob.cancel(); the new await
reads the same lists while the server is live. snapshotSinks copies each under its
own monitor, and the awaited and asserted conditions now share one helper so they
cannot drift apart.
Note `Duration.scaled` is internal to socket-testsuite and NOT visible here despite
the shared package; testTimeScale() is public and documented for exactly this. The
bare delay()s meant CI's QUIC_TEST_TIME_SCALE=3 bought this test nothing.
Verified on JDK 21: 11/11 runs green, and the test now takes 119-133ms instead of
400ms+ of pure sleep. Proven to actually execute rather than vacuously pass, twice:
tightening the assertion to `>= 3` fails with "got 2" (so established == 2 is real,
not absent), and forcing both awaits to an unsatisfiable want=3 makes the test take
10.5s -- two bounded 5s deadlines -- proving both call sites are reached and that
the swallowed timeout falls through to the real assertions.
The issue title names publicOptIn_records_quic_traffic_and_taps_the_network_monitor,
but the failing assertion is in server_capture_mints_a_fresh_sink_per_accepted_connection;
the issue body analyses the correct test.
* ci(linux): build the natives once, then fan the rest of the job out
Same treatment as build-apple, applied to the other monolith. build-linux's `build`
job ran ~28-31 min plus a 6-7 min fuzz tail: build the Linux/Android/Windows quiche
natives, then the JVM/JS/Android checks, then three quiche JVM suites, then link,
then publish, then the JDK-25 multi-release link check.
natives --+-- checks-tests (JVM / JS / Android, keeps the Docker harness)
+-- quiche-jvm (matrix: JNI | FFM | JDK 17/JNI)
+-- link --+-- integration-linux-x64
| +-- integration-linux-arm64
+-- publish
+-- malloc-check, fuzz-jvm, fuzz-native, fuzz-boringssl-asan
review.yaml and merged.yaml need no changes: they reference the reusable workflow,
never its internal job names, and build-linux declares no outputs. The release gate
is unchanged for the same reason it was on Apple — a reusable-workflow call
completes only when all of its jobs complete and fails if any fail.
THE FAN-OUT WOULD HAVE SHIPPED A REAL BUG WITHOUT THIS: `link` now builds
quicEchoJar BEFORE linking. Only cert.crt and cert.key are committed under
socket-quic-quiche/testcerts/; localhost.* and pinned* are gitignored and produced
by generateLocalhostCert/generatePinnedW3cCerts, which are wired via a task-name
regex (build.gradle.kts:2797) matching `(jvm|linuxX64|...)\w*Test` and
`*ProcessResources*Test*`. `linkDebugTestLinuxX64` matches NEITHER. In the monolith
the certs already existed because the test steps had run earlier in the same job;
fanned out, `link` on a fresh runner would have uploaded a testcerts/ containing
two files instead of ten, and every cert-reading quiche test in both integration
jobs would have failed with "Test cert not found". quicEchoJar dependsOn
jvmTestProcessResources, which does match, so building it first restores the
invariant the monolith got by accident.
`natives` also runs buildJvmJniShimLinux{X64,Arm64} through Gradle. The CI shell
script hand-builds the shims and never writes the `.jni-built-<ver>` marker; in one
job that only meant Gradle relinked them later in the same place, but across jobs a
missing marker makes every consumer that touches prepareQuicheNativeLib relink,
including the arm64 cross leg. Running the real task also keeps
-fno-omit-frame-pointer and the strip, so the gdb post-mortems still name
quiche_jni frames.
Linux markers differ from Apple's and were confirmed against the build script, not
assumed: `.built-<ver>-qlog-jvm-<digest>` for BOTH the .so and the .a (one cargo
build emits both, so there is no separate static task and no bare `-qlog` variant),
plus `.jni-built-<ver>`. Android deliberately has no marker at all
(build.gradle.kts:1384: "Do NOT reintroduce a marker + onlyIf gate") and relies on
Gradle's content hashing. All are dotfiles, so the natives upload sets
include-hidden-files: true and the composite action hard-fails if none restore.
Harness colocation, the easiest way to break this file — a stack brought up in one
job is unreachable from another:
* checks-tests keeps the harness certs, the OS/JVM CA trust, the aioquic
HTTP/3+WebTransport server and the ls-qpack QPACK oracle, because
:socket-http3:jvmTest runs the Docker interop tests inside its Gradle call. It
also keeps both netns lanes, which exec a test.kexe that same call builds and
nothing uploads.
* quiche-jvm gets its own certs + CA trust: :socket-quic-quiche:jvmTest
dependsOn(harnessUp), so each leg brings the stack up itself. The JVM-keystore
import still targets $JAVA_HOME only, so the JDK-17 leg still does not see it —
unchanged from before, and left alone deliberately.
The JDK-17 leg keeps its one-setup-java-call-not-two invariant: the LAST entry of a
multi-line java-version becomes JAVA_HOME, so installing 21 and 17 as separate
steps would silently hand Gradle a 17 launcher. It is now an input on the composite
action, off for the two legs that do not need it.
Artifact names and paths are unchanged (quiche-linux-natives, quiche-windows-natives,
android-native-libs, quic-echo-docker-context, linux-{x64,arm64}-test-binary,
maven-local-linux, quic-audit-*, test-reports-linux, hang-threaddump-linux,
quiche-build-logs-linux, fuzz-crashes-*), and all cache keys are byte-for-byte
identical to native-deps-freshness.yaml's mirror of them. Two new names only:
quiche-linux-natives-full for the fan-out, and a per-leg test-reports suffix so the
three parallel quiche-jvm legs cannot collide.
Known, pre-existing, NOT fixed here: setup-linux-fuzz is a second read-write cache
writer on gradle-linux-*/boringssl-*, so it still races `natives` on a miss, and its
quiche keys (-v7-, cargo -v2-) are stale against what build-linux writes (-v8-, -v3-)
and are therefore guaranteed misses. Filed separately rather than widened into this.
Validated: ruamel round-trip clean on all 14 workflows + all 3 composite actions;
actionlint 32 findings on main, 31 here, zero new. Mutation-tested both directions —
deleting a required `with: version` input surfaces
"input \"version\" is required by ... [workflow-call]", and a typo'd `needs:` surfaces
"needs job \"linkk\" which does not exist [job-needs]". Step inventory diffed old vs
new: the only removals are the three JVM test steps that became one matrix leg and
the JDK setup that moved into the composite.
* ci: warm the Apple cache on main, stop losing the klib cache, split the fat K/N lane
Three wall-clock fixes, found by reading step-level timings from the first two
fan-out runs rather than guessing. The fan-out itself was NOT the cost: per-job
overhead measured 25-35s (the composite action: JDK + three cache restores +
artifact download + verify), not the ~3-8 min assumed when the shards were drawn.
Of build-apple's 24m15s, 514s was one avoidable cargo rebuild and 827s was a
single overweight lane.
1. WARM THE APPLE CACHE ON main (~8.5 min off every Apple PR run)
native-deps-freshness.yaml warmed only the Linux triples, so refs/heads/main
carried NO quiche-apple-v3-* entry at all. GitHub scopes cache writes to the
ref that made them — a PR reads its base branch's caches, but one PR cannot
read another's — so with main cold, EVERY Apple PR paid a full from-source
cargo + BoringSSL build for five targets.
Measured on this PR, same content, cold vs warm:
run 31500984797 cold: `Build quiche Apple native libs` 514s, natives 9m48s
run 31505526881 warm: natives 1m32s
One scheduled macOS job a day buys that on every run. Keys/paths are
byte-for-byte identical to build-apple.yaml's natives job, and the quiche entry
keeps its deliberate no-restore-keys rule (#260/#263: a prefix fallback can
resurrect a sibling built from a different source patch).
2. STOP LOSING THE KOTLIN/NATIVE BUILD CACHE (a regression this PR introduced)
Making `natives` the single cache writer was right for avoiding upload races,
but `natives` only runs cargo and a Gradle configure — it compiles no klibs. So
the gradle-macos-* entry it saved held none either: measured 258MB, against
1088MB on a pre-fan-out PR. Every shard was recompiling Kotlin/Native cold on
every run, quietly giving back much of what the parallelism bought.
The widest K/N shard now saves a second key, gradle-macos-kn-*, that actually
contains build-cache-1 output, and the composite action prefers it
(gradle-macos-kn-<hash> -> gradle-macos-kn- -> gradle-macos-). Deliberately not
per-shard keys: five ~1GB entries would blow the 10GB cap and cause exactly the
LRU eviction behind problem 1. Other shards still get the shared commonMain
metadata and dependency resolution even though their own target's klibs are not
in it — partial, and honestly documented as such.
3. SPLIT THE 827s LANE
`macosX64Test ktlintCheck` was 827s of an 864s job. socket-quic-quiche carries
the real QUIC loopback suites and dominates, so it gets its own shard; ktlint
moves to the compile-only ILP32 lane.
⚠️ The split uses `-x`, NOT a hand-kept list of qualified :module:macosX64Test
paths, and that is the whole point. An explicit list is precisely the false
green this file already warns about for `:macosX64Test` vs `macosX64Test`: add
a module, or an Apple target to an existing one, and its tests silently stop
running while everything stays green. `-x` inverts the failure mode — the
unqualified task still means "every subproject that has one", so a new module
lands in the `rest` shard automatically. The worst case is running something
twice, which is loud and cheap.
Verified by dry-run, not assumed: unqualified macosX64Test resolves to 10
modules (root, network-monitor, socket-http3, socket-quic, socket-quic-default,
socket-quic-quiche, socket-testkit, socket-testsuite, socket-udp,
socket-webtransport); with `-x :socket-quic-quiche:macosX64Test` it resolves to
exactly those 10 minus that one.
Validated: ruamel round-trip clean on all 14 workflows + all 3 composite actions;
actionlint 32 findings on main, 31 here, zero new.
* ci(apple): resize the FFM timeout the split invalidated, and rebalance the K/N lanes
Run 3 went red on `JVM quiche (FFM)`. It was not a hang, an assertion, or a
regression — the step ran 733s against a `timeout-minutes: 12` (720s) and was
killed. Measured across three runs on equivalent content: 618s, 637s, 733s.
That limit was mis-sized BY the fan-out. The monolith gave this step 8 minutes
because it ran AFTER the JNI step in the SAME job, so compilation was already done
and the Gradle daemon warm. Fanned out, each backend lane compiles from cold, and I
carried over a number that no longer described the work. Raised to 20 min, with the
job-level timeout raised to 30 so a genuine hang is still bounded and still names
the step faster than the job timeout would.
REBALANCE — the previous commit's split moved the bottleneck instead of removing it:
lane run 2 run 3
macOS x64 + ktlint 978s split -> 563s / 522s (worked)
device-compile (+ktlint) 414s 813s (new bottleneck)
tvOS + watchOS 833s 791s
Splitting macosX64Test was right, but parking ktlintCheck on the compile lane just
moved ~400s of work onto the next-shortest lane and made IT the gate. ktlint now has
its own lane: it depends on no native artifact and no test, so it parallelises
freely, and at ~35s per-job overhead a dedicated lane is nearly free.
tvOS and watchOS also split apart (791-833s together, the second-longest lane).
Neither has a quiche target, so their QUIC suites skip via UnsupportedQuicEngine and
what remains is mostly compile, which parallelises well.
Expected longest K/N lane ~658s (iOS simulator) against 978s before this pair of
commits. Stated as expected, not measured — the last three shard estimates I made
without step-level timings were all wrong, most recently the ktlint placement in the
commit directly before this one.
Validated: ruamel round-trip clean; actionlint 32 on main, 31 here, zero new.
1 parent 4aca8d1 commit 1efafc4
6 files changed
Lines changed: 1715 additions & 536 deletions
File tree
- .github
- actions
- apple-quiche-env
- linux-quiche-env
- workflows
- socket-quic-quiche/src/jvmTest/kotlin/com/ditchoom/socket/quic
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
| 1 | + | |
| 2 | + | |
| 3 | + | |
| 4 | + | |
| 5 | + | |
| 6 | + | |
| 7 | + | |
| 8 | + | |
| 9 | + | |
| 10 | + | |
| 11 | + | |
| 12 | + | |
| 13 | + | |
| 14 | + | |
| 15 | + | |
| 16 | + | |
| 17 | + | |
| 18 | + | |
| 19 | + | |
| 20 | + | |
| 21 | + | |
| 22 | + | |
| 23 | + | |
| 24 | + | |
| 25 | + | |
| 26 | + | |
| 27 | + | |
| 28 | + | |
| 29 | + | |
| 30 | + | |
| 31 | + | |
| 32 | + | |
| 33 | + | |
| 34 | + | |
| 35 | + | |
| 36 | + | |
| 37 | + | |
| 38 | + | |
| 39 | + | |
| 40 | + | |
| 41 | + | |
| 42 | + | |
| 43 | + | |
| 44 | + | |
| 45 | + | |
| 46 | + | |
| 47 | + | |
| 48 | + | |
| 49 | + | |
| 50 | + | |
| 51 | + | |
| 52 | + | |
| 53 | + | |
| 54 | + | |
| 55 | + | |
| 56 | + | |
| 57 | + | |
| 58 | + | |
| 59 | + | |
| 60 | + | |
| 61 | + | |
| 62 | + | |
| 63 | + | |
| 64 | + | |
| 65 | + | |
| 66 | + | |
| 67 | + | |
| 68 | + | |
| 69 | + | |
| 70 | + | |
| 71 | + | |
| 72 | + | |
| 73 | + | |
| 74 | + | |
| 75 | + | |
| 76 | + | |
| 77 | + | |
| 78 | + | |
| 79 | + | |
| 80 | + | |
| 81 | + | |
| 82 | + | |
| 83 | + | |
| 84 | + | |
| 85 | + | |
| 86 | + | |
| 87 | + | |
| 88 | + | |
| 89 | + | |
| 90 | + | |
| 91 | + | |
| 92 | + | |
| 93 | + | |
| 94 | + | |
| 95 | + | |
| 96 | + | |
| 97 | + | |
| 98 | + | |
| 99 | + | |
| 100 | + | |
| 101 | + | |
| 102 | + | |
| 103 | + | |
| 104 | + | |
| 105 | + | |
| 106 | + | |
| 107 | + | |
| 108 | + | |
| 109 | + | |
| 110 | + | |
| 111 | + | |
| 112 | + | |
| 113 | + | |
| 114 | + | |
| 115 | + | |
| 116 | + | |
| 117 | + | |
| 118 | + | |
| 119 | + | |
| 120 | + | |
| 121 | + | |
| 122 | + | |
| 123 | + | |
| 124 | + | |
| 125 | + | |
| 126 | + | |
| 127 | + | |
| 128 | + | |
| 129 | + | |
| 130 | + | |
| 131 | + | |
| 132 | + | |
| 133 | + | |
| 134 | + | |
| 135 | + | |
0 commit comments