Repository navigation
export buildkit history and structured lifecycle telemetry at teardown (release/v2) - #128
Conversation
🤖 Devin AI EngineerI'll be helping with this pull request! Here's what you should know: ✅ I will automatically:
Note: I can only respond to comments from users who have write access to this repository. ⚙️ Control Options:
|
5ce4997 to
303b922
Compare
| // Export this job's build history (raw record + trace bytes) and delete | ||
| // the records afterwards so history.db and its content-store attachments | ||
| // never get committed to the sticky disk. Must happen before shutdown. | ||
| const builds = await exportBuildHistory(buildkitdAddr, lifecycle); |
There was a problem hiding this comment.
is this valid? The sticky disk as a block io device doesn't know about which logical files have been deleted, right?
There was a problem hiding this comment.
Right — the block device has no notion of file deletion, so residual bytes can remain in freed blocks after commit; we don't claim physical scrubbing. What the delete does guarantee is filesystem-level: it happens via buildkitd's UpdateBuildHistory{Delete} API (BoltDB record delete + content-store blob unref) before buildkitd shutdown → integrity check → unmount → commit, so the committed filesystem no longer references those records. The purpose is (1) history.db doesn't grow unboundedly across runs and (2) the next job's export doesn't re-export prior jobs' records — not data erasure. I've reworded the comment to say exactly that (was misleadingly phrased as "never get committed").
303b922 to
87c7d43
Compare
|
@devin do we bound the size of the report we generate in terms of bytes? I think we need a way to hard bound it somewhere around 10s of MiB and under 128 MiB |
|
Yes — the report is hard-bounded to ~10 MiB worst case, enforced at assembly time in
Everything else in the report (lifecycle facts, du totals, per-cache-mount rows) is small fixed-size fields, so the whole |
87c7d43 to
b357178
Compare
bc50154 to
0ccc66c
Compare
devin-ai-integration
Bot
changed the title
0ccc66c to
65828a3
Compare
At post-step teardown, export the job's raw BuildKit history records (plus solve-status traces, logs excluded, size-capped) via ListenBuildHistory, delete them so history.db and its content-store attachments are never committed to the sticky disk, and ship them with the runner _diag step timeline and structured lifecycle facts (builder mode/fallback reason, commit decision/skip reason, buildctl du totals incl. per-cache-mount, fs usage, setup/shutdown timings) to the vm-agent's ReportDockerBuild RPC over the shared agent client (BLACKSMITH_AGENT_ADDR discovery). docker_build_ids are issued host-side; all reporting is fail-soft with bounded timeouts and never fails the customer job, and is skipped silently when no agent is reachable or it predates the RPC. BuildKit control and containerd content protos are vendored verbatim from pinned upstream releases (scripts/fetch-protos.sh) and compiled with protoc-gen-es into src/gen; generated code is excluded from eslint. Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
A whole-job report can exceed the vm-agent's default 4 MiB gRPC MaxRecvMsgSize (a 24-build job produces a ~6 MB message that is rejected with resource_exhausted, dropping all telemetry). Builds are split across multiple unary ReportDockerBuild calls with a ~3 MiB per-request budget covering the runner step timeline plus record and trace bytes. The timeline ships in every chunk (workflow-step attribution is per-request server-side); the lifecycle message ships only in the first chunk so the agent never double-emits lifecycle metrics. Per-record/trace caps and fail-soft behavior are unchanged, and each chunk gets its own report timeout. Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
65828a3 to
fb31cd1
Compare
Summary
Targets
release/v2only. At teardown, the action exports the job's BuildKit build history and ships it — with structured lifecycle telemetry — to the vm-agent'sReportDockerBuildRPC over the shared v2 agent client (src/reporter.ts,BLACKSMITH_AGENT_ADDRdiscovery). The guest ships raw bytes; all semantic parsing lives host-side in the fa agent so BuildKit format drift is absorbed by an agent deploy, not an action release.What ships
Control.ListenBuildHistory({EarlyExit:true})streams the job'sBuildHistoryRecords. Each record is decoded as a typed protobuf-es message only to readRef(for post-export deletion),CompletedAt(incomplete flag), and the trace descriptor, and to clearlogs; it is then re-serialized and shipped opaquely (protobuf-es preserves unknown fields across the round trip, so fields newer than our vendored proto still reach the agent). Exported records are deleted viaControl.UpdateBuildHistory(Delete:true)so telemetry state is not committed into the sticky disk.Content.Read(digest), size-capped (2 MiB per trace; oversized → shipped without trace, markedtruncated)._diag/Worker_*.logread verbatim, tail-capped at 1 MiB; parsing is agent-side.buildctl du -vcache breakdown (per-mount sizes), fs used/size, hotload / buildkitd-ready / buildkitd-shutdown timers, SIGKILL flag, history export-timeout and history-delete-failed flags.v2 semantics
integrityOutcome = SKIPPED,integrityDurationMs = 0: v2 has no bbolt integrity check.pruneTriggered = false,pruneBytes = 0: v2 relies on native GC; there is no explicit cache prune at teardown. History-record deletion is tracked separately (historyPruneFailed).STEP_FAILURES/AMBIGUOUS/SIGKILL/CLEANUP_ERROR/NO_EXPOSEmap onto the existingCommitSkipReasonvalues; a successfulcommitStickyDiskcall recordsCommitDecision.REQUESTED. The v2-only outcomes use the dedicatedCommitSkipReasonvalues from the bumped@buf/blacksmith_vm-agent.bufbuild_espackage: mount-timecommitEarlyDenyReasonandPrepareCommitshouldCommit=false→SERVER_DECLINED; a pre-commit hook asking to skip →HOOK_SKIPPED. Against an agent that predates these enum values the row still inserts (commit_decision=skipped, empty skip reason); an agent withoutReportDockerBuildreturnsUnimplemented, which is treated as unsupported and skipped.state-helpersince main and post are separate processes.Chunked reporting
A whole-job report can exceed the vm-agent's default 4 MiB gRPC
MaxRecvMsgSize(a 24-build job produces a ~6 MB message rejected withresource_exhausted).reportDockerBuildsplits builds across multiple unaryReportDockerBuildRequestcalls viachunkBuilds, with a ~3 MiB per-request budget covering timeline + record + trace bytes:Protos
BuildKit
control.proto(+ deps) and containerdcontent.protoare vendored verbatim from pinned upstream releases (buildkit v0.32.2, containerd v2.3.4, googleapis pinned commit) viascripts/fetch-protos.shand generated withbuf generate→src/gen/(ignored by eslint).stickydisk.prototypes come from the@buf/blacksmith_vm-agent.bufbuild_espackagerelease/v2already depends on (which includesReportDockerBuild). Generation is dev-time only:pnpm generate→pnpm build→ checked-indist/. Nothing is fetched or compiled during an action run, and no hand-authored proto content or hand-rolled wire parsing remains.Ordering
Post-step teardown:
buildctl du -vsnapshot → history export → history-record delete → buildkitd shutdown → fs usage → unmount → commit/skip decision → chunked report.Identity
The guest sends only
expose_id,vm_id, git sha/branch, and build facts; keys, entity, run/job attribution are stamped host-side by the vm-agent.Companion PRs: agent-side parsing/insertion FastActions/fa#5232 + OTLP JSON trace parser FastActions/fa#5277 (both merged); ClickHouse schema useblacksmith/web#11965 (merged); v2 skip reasons FastActions/fa#5441 (merged).
Need help on this PR? Tag
@codesmith-botwith what you need. Autofix is disabled.Link to Devin session: https://app.devin.ai/sessions/0c6ebc2a6de54933bebe821e21ab8b76
Open in Devin Desktop: https://app.devin.ai/desktop/session/0c6ebc2a6de54933bebe821e21ab8b76?variant=devin
Requested by: @piob-io