Repository navigation
Commit f19948f
committed
feat(compressor): add tracing instrumentation for scheme selection
The compressor previously had no structured observability into which schemes
were chosen, how accurate their estimators were, or when a picked scheme
produced a larger-than-canonical output. This change adds a focused set of
tracing events so those questions can be answered with a short `jq` pipeline
against the JSON log stream.
What is emitted, under three targets:
`vortex_compressor::cascade` — `compress` span (rollup fields on exit),
`cascade_exhausted` event.
`vortex_compressor::encode` — `scheme.compress_result` per leaf decision,
`scheme.compress_failed` on error.
`vortex_compressor::estimate` — `sample.result` per sampled scheme,
`sample.compress_failed` on error.
The primary event is `scheme.compress_result`, which carries `scheme`,
`before_nbytes`, `after_nbytes`, `estimated_ratio` (absent for `AlwaysUse`
winners), `actual_ratio`, and `accepted`. Per-scheme savings, rejection counts,
and estimator accuracy are all derivable from this one event.
`scheme.compress_failed` and `sample.compress_failed` are new ERROR events that
surface third-party scheme errors with the scheme id + error message before
the VortexResult propagates, instead of the error silently losing compressor
context.
Orthogonal refactor bundled in (kept from the original PR): the
`CompressionEstimate` enum is split into a terminal `EstimateVerdict`
(Skip | AlwaysUse | Ratio) and a `DeferredEstimate` (Sample | Callback). The
`EstimateFn` callback now returns `EstimateVerdict` directly, which statically
forbids the previously-panicking Sample/Estimate return cases.
Minor related cleanups in `vortex-btrblocks`: add a pointer from its docs to
the compressor observability docs, and drop a stray `tracing::debug!` call
in ZigZag that predated any structured compressor tracing.
Field names should be considered unstable — expect them to change as we gain
experience with what is actually useful.
Tracking issue: #7216
Signed-off-by: Claude <[email protected]>1 parent ce52b71 commit f19948f
9 files changed
Lines changed: 500 additions & 121 deletions
File tree
- vortex-btrblocks/src
- schemes
- vortex-compressor
- src
- tests
Some generated files are not rendered by default. Learn more about customizing how changed files appear on GitHub.
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
38 | 38 | | |
39 | 39 | | |
40 | 40 | | |
| 41 | + | |
| 42 | + | |
| 43 | + | |
| 44 | + | |
41 | 45 | | |
42 | 46 | | |
43 | 47 | | |
| |||
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
52 | 52 | | |
53 | 53 | | |
54 | 54 | | |
| 55 | + | |
| 56 | + | |
| 57 | + | |
| 58 | + | |
| 59 | + | |
| 60 | + | |
| 61 | + | |
| 62 | + | |
| 63 | + | |
| 64 | + | |
| 65 | + | |
| 66 | + | |
| 67 | + | |
55 | 68 | | |
56 | 69 | | |
57 | 70 | | |
| |||
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
298 | 298 | | |
299 | 299 | | |
300 | 300 | | |
301 | | - | |
302 | | - | |
303 | 301 | | |
304 | 302 | | |
305 | 303 | | |
| |||
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
19 | 19 | | |
20 | 20 | | |
21 | 21 | | |
22 | | - | |
| 22 | + | |
23 | 23 | | |
24 | 24 | | |
25 | 25 | | |
| |||
29 | 29 | | |
30 | 30 | | |
31 | 31 | | |
| 32 | + | |
32 | 33 | | |
33 | 34 | | |
34 | 35 | | |
| |||
0 commit comments