You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
{{ message }}
Repository navigation
Commit c23468e
Browse filesBrowse the repository at this point in the historyBrowse files
fix: cover every pool type in the total, and pair trace samples
Addresses review feedback on the process-wide reservation counter.
Source the total from the per-thread tracing registry instead of the
task-shared one. The task-shared registry only holds pools created by
`acquire_task_shared_pool`, so the `greedy`, `fair_spill`, the `_global`
variants and `unbounded` modes reported zero while holding live
reservations, and the analyzer then called those reservations untracked.
Every plan registers whichever pool `create_memory_pool` gave it, so
deduplicating that registry by pool identity covers all of them and still
counts a shared pool once.
That also removes the `Weak::upgrade` under the registry lock. An
upgraded reference could become a pool's last owner if the owning task
finished mid-scan, and `TaskSharedMemoryPool::drop` re-locks the
registry, so the sampling thread would have deadlocked holding it. The
thread registry holds strong references that the scan only borrows.
In the analyzer, compare an allocation sample only against the pool total
emitted next to it on the same thread. `executePlan` emits the two back to
back, but a busy-poll reservation event in between meant the previous
comparison paired a fresh allocation with a stale total and recorded
excess that was never held. Traces without the total counter still take
the legacy path, whose numbers are unchanged. A trace that carries the
counter but never pairs a sample now says so rather than reporting OK.
| jvm_heap_used | JVM heap memory usage of live objects for the executor process |
130
-
| jemalloc_allocated | Native memory usage for the executor process (requires `jemalloc` feature) |
131
-
| native_allocated | Bytes handed out by the Rust global allocator, process-wide (requires `alloc-accounting` feature). Approximate to within 64 KiB of un-flushed delta per live thread. |
132
-
| comet_memory_reserved_total | Total memory reserved across every live Comet memory pool, process-wide. Counts each task-shared pool once, so unlike the per-thread counters it can be compared directly against an allocation counter.|
133
-
| thread_NNN_comet_memory_reserved | Memory reserved by Comet's DataFusion memory pool (summed across all contexts on the thread). NNN is the Rust thread ID. Do not sum these across threads: a task-shared pool reports its full reservation on every thread that references it. |
134
-
| thread_NNN_comet_jvm_shuffle | Off-heap memory allocated by Comet for columnar shuffle. NNN is the Rust thread ID. |
| jvm_heap_used | JVM heap memory usage of live objects for the executor process |
136
+
| jemalloc_allocated | Native memory usage for the executor process (requires `jemalloc` feature) |
137
+
| native_allocated | Bytes handed out by the Rust global allocator, process-wide (requires `alloc-accounting` feature). Approximate to within 64 KiB of un-flushed delta per live thread. |
138
+
| comet_memory_reserved_total | Total memory reserved across every live Comet memory pool, process-wide, whatever the configured pool type. Counts a pool shared between execution contexts once, so unlike the per-thread counters it can be compared directly against an allocation counter. |
139
+
| thread_NNN_comet_memory_reserved | Memory reserved by Comet's DataFusion memory pool (summed across all contexts on the thread). NNN is the Rust thread ID. Do not sum these across threads: a shared pool reports its full reservation on every thread that references it.|
140
+
| thread_NNN_comet_jvm_shuffle | Off-heap memory allocated by Comet for columnar shuffle. NNN is the Rust thread ID. |
0 commit comments