Skip to content

Commit f19948f

Browse files
committed
feat(compressor): add tracing instrumentation for scheme selection
The compressor previously had no structured observability into which schemes were chosen, how accurate their estimators were, or when a picked scheme produced a larger-than-canonical output. This change adds a focused set of tracing events so those questions can be answered with a short `jq` pipeline against the JSON log stream. What is emitted, under three targets: `vortex_compressor::cascade` — `compress` span (rollup fields on exit), `cascade_exhausted` event. `vortex_compressor::encode` — `scheme.compress_result` per leaf decision, `scheme.compress_failed` on error. `vortex_compressor::estimate` — `sample.result` per sampled scheme, `sample.compress_failed` on error. The primary event is `scheme.compress_result`, which carries `scheme`, `before_nbytes`, `after_nbytes`, `estimated_ratio` (absent for `AlwaysUse` winners), `actual_ratio`, and `accepted`. Per-scheme savings, rejection counts, and estimator accuracy are all derivable from this one event. `scheme.compress_failed` and `sample.compress_failed` are new ERROR events that surface third-party scheme errors with the scheme id + error message before the VortexResult propagates, instead of the error silently losing compressor context. Orthogonal refactor bundled in (kept from the original PR): the `CompressionEstimate` enum is split into a terminal `EstimateVerdict` (Skip | AlwaysUse | Ratio) and a `DeferredEstimate` (Sample | Callback). The `EstimateFn` callback now returns `EstimateVerdict` directly, which statically forbids the previously-panicking Sample/Estimate return cases. Minor related cleanups in `vortex-btrblocks`: add a pointer from its docs to the compressor observability docs, and drop a stray `tracing::debug!` call in ZigZag that predated any structured compressor tracing. Field names should be considered unstable — expect them to change as we gain experience with what is actually useful. Tracking issue: #7216 Signed-off-by: Claude <[email protected]>
1 parent ce52b71 commit f19948f

9 files changed

Lines changed: 500 additions & 121 deletions

File tree

‎Cargo.lock‎

Lines changed: 1 addition & 0 deletions
Some generated files are not rendered by default. Learn more about customizing how changed files appear on GitHub.

‎vortex-btrblocks/src/canonical_compressor.rs‎

Lines changed: 4 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -38,6 +38,10 @@ pub struct BtrBlocksCompressor(
3838

3939
impl BtrBlocksCompressor {
4040
/// Compresses an array using BtrBlocks-inspired compression.
41+
///
42+
/// This is a thin delegate to [`CascadingCompressor::compress`], which owns the tracing
43+
/// instrumentation. See the `Observability` section of the [`vortex_compressor`] crate
44+
/// docs for the full tracing reference.
4145
pub fn compress(&self, array: &ArrayRef) -> VortexResult<ArrayRef> {
4246
self.0.compress(array)
4347
}

‎vortex-btrblocks/src/lib.rs‎

Lines changed: 13 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -52,6 +52,19 @@
5252
//! .build();
5353
//! ```
5454
//!
55+
//! # Observability
56+
//!
57+
//! [`BtrBlocksCompressor`] is a thin delegate to [`CascadingCompressor`], which owns all
58+
//! tracing instrumentation. See the
59+
//! [`vortex_compressor` crate docs](vortex_compressor#observability) for the full reference
60+
//! on targets, spans, and events.
61+
//!
62+
//! Quick start: one line per leaf with scheme, estimated ratio, actual ratio, accepted?:
63+
//!
64+
//! ```text
65+
//! RUST_LOG=vortex_compressor::encode=debug cargo test -p vortex-btrblocks
66+
//! ```
67+
//!
5568
//! [BtrBlocks]: https://www.cs.cit.tum.de/fileadmin/w00cfj/dis/papers/btrblocks.pdf
5669
5770
mod builder;

‎vortex-btrblocks/src/schemes/integer.rs‎

Lines changed: 0 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -298,8 +298,6 @@ impl Scheme for ZigZagScheme {
298298

299299
let compressed = compressor.compress_child(&encoded.into_array(), &ctx, self.id(), 0)?;
300300

301-
tracing::debug!("zigzag output: {}", compressed.encoding_id());
302-
303301
Ok(ZigZag::try_new(compressed)?.into_array())
304302
}
305303
}

‎vortex-compressor/Cargo.toml‎

Lines changed: 2 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -19,7 +19,7 @@ num-traits = { workspace = true }
1919
parking_lot = { workspace = true }
2020
rand = { workspace = true }
2121
rustc-hash = { workspace = true }
22-
tracing = { workspace = true }
22+
tracing = { workspace = true, features = ["std", "attributes"] }
2323
vortex-array = { workspace = true }
2424
vortex-buffer = { workspace = true }
2525
vortex-error = { workspace = true }
@@ -29,6 +29,7 @@ vortex-utils = { workspace = true }
2929
[dev-dependencies]
3030
divan = { workspace = true }
3131
rstest = { workspace = true }
32+
tracing-subscriber = { workspace = true, features = ["env-filter"] }
3233
vortex-array = { workspace = true, features = ["_test-harness"] }
3334

3435
[lints]

0 commit comments

Comments
 (0)