docs(observe): per-request ingestion limits, busy refusals and collector settings [TH-8745] - #1029

Stacked on #1028. Keep draft and on the existing base; publish after the collector release and retarget only after the parent merges.

Updated pages

• observe/reference/export-formats.mdx: request/body/work-pool limits, exact ClickHouse JSONEachRow-before-compression terminology, project-name validation, transient project recovery, partial-success drops, quota exporter differences, and the new 30-second body/message deadlines. The corrected Success row and linked project-failure section explain that an OTLP Export response may be empty on full success, but HTTP 200 / gRPC OK may carry partial_success.rejected_spans and an error message. Inspect that field before treating all submitted spans as accepted; visibility of accepted spans is qualified by batching, storage progress and load.
• observe/troubleshooting/no-traces-appearing.mdx: INVALID_ARGUMENT for oversized gRPC requests, with RESOURCE_EXHAUSTED identified as earlier-version behavior.
• observe/troubleshooting/noisy-or-incomplete-traces.mdx: the same corrected oversized-request status-code guidance.

Standalone guidance distinguishes the 216 MiB total budget from its 144 MiB work pool. Attribute-heavy 8192-span batches may not fit; start Collector send_batch_size/send_batch_max_size at 4096 and adjust from measured traffic. A 15-second exporter timeout can help fitting large batches, but does not make oversized work fit. GOMEMLIMIT is not a hard RSS bound. The 512MiB setting was signed off on 2026-10-25 from collector-only measurements; the page states that whole-Standalone load at minimum memory has not been measured.

Follow-up at 55642150ec36

The export-formats page now distinguishes exact resource names from stricter header transport syntax. Visible variation-selector/ZWJ names work for both new and cached projects; UTF-8, 255-character, Cc and wholly-invisible collector safeguards remain explicit without changing the app create/rename policy.

It also documents retryable authentication-cache expiry/eviction/replacement during project lookup/create: HTTP 503 + Retry-After: 1 / gRPC UNAVAILABLE, no span rows or usage on the failed export, fresh authentication on retry, no stale merge into replacement results and fail-closed revocation. Project creation can already have succeeded before retry.

Metering change — accepted 2026-10-24. The collector uses canonical OTLP protobuf size of accepted, stamped traces across HTTP protobuf/JSON/plain/gzip and gRPC. This preserves gRPC's existing unit but intentionally changes full-success HTTP usage totals too, rather than only removing rejected payload from mixed requests. The unit is accepted semantic protobuf bytes, including accepted resource/scope metadata and stamped attributes, not raw JSON, compressed request bytes, JSONEachRow or compressed table storage. Old HTTP used decompressed-body length; gzip ratio already did not affect it. Event-mode units, enqueue timing and retry/deduplication semantics are unchanged.

Nikhil accepted this unit for both transports on 2026-10-24, as implemented. No release readiness is claimed: keep this PR draft and publish only after the collector release. F2/F3 are inherited from the code stack parent, not regressions introduced by the prior child repair.

Fresh verification of the exact source 55642150ec36fa465b5712dacc8431d9032f31be: isolated npm run build passed in 62.891s using a copied dependency tree and 3072MiB Node cap; npm run audit-links passed in 0.331s (720 pages, 687 nav entries, zero broken navigation/content links, 29 existing orphans). All 1826 tracked build inputs match the worktree/commit, and rendered HTML confirms naming, cache-retry, full-success HTTP and then-pending decision wording plus the new usage-unit anchor. Existing Pagefind/no-HTML and Node deprecation warnings remain. These are local tests, not hosted CI, live external-URL checks or application E2E coverage. Collector companion: 75c2d93838e4df3d9c912995b42d95fcbc0be938 / future-agi#3598.

Verification of 43fb64ee7c2534073f8cb3ecc8d1084e970d2a65: isolated npm run build passed in 46.497s using a copied dependency tree and 3072MiB Node cap; npm run audit-links passed in 0.35s (720 pages, 687 nav entries, zero broken navigation/content links, 29 existing orphans). All 1826 tracked build inputs match the commit, and rendered HTML no longer contains the draft metering warning or "proposed" in the usage-unit section and keeps the usage-unit anchor. These are local results, not hosted CI.

Verification of 8f309693bad9d5badf401de7e65c741c3ca9a830 (one sentence in export-formats.mdx: the pending sign-off wording became "has not been measured"): isolated npm run build passed (1181 pages built, Pagefind indexed 716) and npm run audit-links passed (720 pages, 687 nav entries, zero broken navigation/content links, 29 existing orphans). The rendered page contains the new sentence and not the old one. Local checks, not hosted CI.

Historical verification at 2af57acd

The exact three-page source delivered in 2af57acd48e21ce46f953cc823b36095900a360f was checked locally after the partial-success correction:

• npm run build passed in 54.480s in a fresh external scratch copy using the existing isolated dependencies, copied rather than symlinked.
• npm run audit-links passed in 0.311s: 720 pages, 687 navigation entries, zero broken navigation/content-page links and 29 existing orphan pages. This is an internal page-link audit, not a live external-URL check.
• Rendered HTML checks confirm the new project-failure anchor, full-success caveat, rejected-span inspection instruction and qualified visibility timing. git diff --check passed; committed file hashes match the build snapshot.

The build retains the existing Pagefind warning about 465 pages without an outer <html> element and a Node module.register() deprecation warning. The earlier repair's first scratch build failed because a cross-volume node_modules symlink confused Astro's compiled-module cache; its isolated dependency copy then passed. This delivery's fresh build passed without that failure. These are local results, not hosted-CI evidence; any push-triggered hosted checks must be read at the delivered head separately.

The generated self-hosting configuration-reference page remains generated from the code repository rather than edited here.

Persistent upstream recipe at 6a7f3bff

Added a linked Observe guide plus downloadable otelcol-contrib 0.163.2 HTTP/gRPC configs pinned by image digest. Default SDK guidance is unchanged. The existing PR check now also runs for this PR's stacked base; no checks were removed or relaxed.

• Collector companion frozen at 6834db5dfaf2fa9925f60446ea844ba114b15699; all 186 source files and all 22 SDK pins match the isolated VM inputs. No code-branch edits were made by this lane.
• Same finite burst: 657,288 sent/upstream-ACKed IDs; all 591,752 fitting IDs stored; 65,536 deliberate oversized controls refused; zero fitting loss. No sender batch/concurrency/export-timeout changes. SDK ACKs here are upstream-hop acknowledgements, not backend storage confirmations.
• Collector-only hard 512 MiB, swap disabled: lifetime cgroup peak 362,418,176 bytes, no OOM/restart. Upstream is separate and uncapped (2,996,670,464-byte cgroup peak). Full canonical/catalog drain 1,276.96s. The initial observer timed out; the disclosed 323.13s sampling gap is covered by same-PID kernel lifetime peaks/events, not a claimed continuous sample record.
• Actual mid-drain SIGKILL/restart: all 32,768 acknowledged IDs persisted. Logical persistent-queue overflow: HTTP 503 / gRPC UNAVAILABLE clearly rejected 1,920 IDs per case; all 1,152 acknowledged IDs per case stored after recovery.
• Extra physical-storage-full control FAILS: forcing a 2 MiB file-storage growth cap acknowledged 896 IDs and clearly refused 2,176, but 640 previously acknowledged IDs remain missing after restoring headroom/restarting. The page prominently discloses this unsafe upstream boundary. Do not call the stronger disk-exhaustion durability requirement cleared or accept this loss as normal. No release/merge clearance is granted by the finite-burst pass.
• Final source docs build (npm ci, npm run build) and audit PASS on the staging VM: 721 pages discovered, 688 navigation entries, zero broken navigation/content links, 29 inherited orphans. Exact downloadable configs match those tested; rendered page and public assets exist.
• Exact-head hosted PR Checks 40595314553 SUCCESS (link audit, lock sync and deleted-page redirects). This hosted workflow does not run the full Astro build; that was executed separately on the VM.

Keep this PR DRAFT, stacked on #1028, and unmerged until the collector release and required review. Physical-storage-exhaustion acceptance remains a separate blocker/decision; do not infer a product release or unconditional reliable-buffer guarantee.

🤖 Generated with Claude Code