build-telemetry: finalize pending history traces, raise the job-wide payload cap, and report export caps in the lifecycle - #135

Summary

Three follow-ups to the post-step history export from #128, each a self-contained commit: builds no longer lose their trace to buildkitd's asynchronous trace finalization, the job-wide payload cap is sized for the chunked reporting that #128 introduced, and every cap that can shrink the telemetry is reported in the job lifecycle so gaps are explainable from ClickHouse.

Finalize pending history records before exporting traces

buildkitd attaches a build's OTLP trace to its history record asynchronously: after emitting the COMPLETE event it keeps the span recorder open for up to 3s (waiting for a finalizer or the timer) before writing the trace blob to the content store and stamping record.trace. A post step that starts right after the last docker build lists a completed record with trace == undefined, ships it without a trace, and then prunes it — the row lands in docker_build_summary with trace_missing=1 and an empty steps array. In an end-to-end lab run this hit the final build of most multi-build jobs and every single-build job that ended just before teardown (69 of 205 rows).

exportBuildHistory runs a pre-pass before collection:

// completed records without a trace descriptor
for await (ev of control.listenBuildHistory({EarlyExit: true}))
  if (rec.CompletedAt !== undefined && rec.trace === undefined) pending.push(rec.Ref)

await Promise.all(pending.map(ref =>
  withTimeout(control.updateBuildHistory({Ref: ref, Finalize: true}), 5_000)))  // fail-soft

UpdateBuildHistory{Finalize:true} triggers the record's finalizer and blocks until the trace is written (no-op for records already finalized), so the collection pass sees the descriptor. Finalize failures/timeouts are logged at debug and the record is still exported (without trace). A lab rerun with this change went from 69/205 to 0/185 trace_missing rows.

Job-wide history payload cap: 32 MiB

MAX_TOTAL_PAYLOAD_BYTES bounds record + trace bytes for the whole job. At 8 MiB it was the single-request bound from before chunked reporting; with ~3 MiB ReportDockerBuild chunks the per-request size is already controlled, so the job-wide cap only decides how many builds of a many-build job keep their traces — at 8 MiB roughly a dozen builds with ~1 MiB traces fit and the rest ship truncated with no steps. 32 MiB lets such jobs ship whole while still bounding teardown. The per-record (512 KiB), per-trace (2 MiB), per-request (~3 MiB) and timeline (1 MiB) caps and the 15s export timeout are unchanged.

Export caps and timeline size reported in DockerJobLifecycle

Until now the only evidence that a cap fired was a core.debug line in the runner log. The lifecycle (sent in the first report chunk only) now carries:

Job-wide admission is factored into admitBuild(build, traceUnavailable, counters) so the three drop counters are disjoint — a build lands in at most one, whole-record drops taking precedence — and unit-tested. Counters accumulate during collection and are copied onto the lifecycle only when the export completes, so a timed-out export (which ships no builds) reports zeros alongside history_export_timed_out. readRunnerStepTimeline(lifecycle) records the shipped size/truncation via capRunnerStepTimeline.

Requires @buf/blacksmith_vm-agent.bufbuild_es 2.14.1-20260904184135-ab292bb91e69.3 (fields added agent-side in FastActions/fa#5449; columns in useblacksmith/web#12482). Agents that predate the fields ignore them; the row still inserts with the columns at their defaults.

dist/ is rebuilt.

^{Need help on this PR? Tag @codesmith-bot with what you need. Autofix is disabled. (Staging)}

^{Need help on this PR? Tag @codesmith-bot with what you need. Autofix is disabled.}

Link to Devin session: https://app.devin.ai/sessions/0c6ebc2a6de54933bebe821e21ab8b76
Open in Devin Desktop: https://app.devin.ai/desktop/session/0c6ebc2a6de54933bebe821e21ab8b76?variant=devin
Requested by: @piob-io