Skip to content

Commit 2bf8895

Browse files
authored
fix: bound the streaming mel buffer to a sliding window (fixes mudler#63, the streaming RSS leak) (mudler#69)
Root cause: parakeet_capi_stream_feed's streaming session keeps every mel frame from stream_begin onward in mel_buf and rebuilds that whole, ever-growing array on every feed call, even though the decoder (feed_available's window()) never reads a frame older than mel_buffer_idx - pre_encode_cache_size() frames back — so those old frames are provably dead the moment the decoder consumes past them. Because each rebuild requests a uniquely, ever-larger allocation for the life of the stream, the allocator can never reuse an earlier call's freed (now permanently too-small) block, so process RSS grows with the cumulative history of buffer sizes rather than the small, bounded set of frames actually still needed — which is why neither stream_free nor a fresh stream_begin reclaims it. The fix: track the absolute frame index of mel_buf's first column (mel_buf_origin) and, on every append, drop the prefix strictly before mel_buffer_idx - pre_encode_cache_size() — the exact bound window() already relies on — so mel_buf stays bounded to O(pre_encode_cache_size + chunk_size) instead of O(stream length). No API change, no behavioural change: window()'s absolute-index reads are translated into mel_buf's origin-relative local columns, and the surviving frames are carried forward byte-for-byte. Measurement (CPU + Accelerate build, realtime_eou_120m-v1 q8, 120 s of silence fed in 20 ms chunks, RSS sampled at 30/60/90/120 s of audio fed): growth over the last 60 s drops from 178 MB to 0.5 MB; the same run's wall time drops from ~14 minutes to ~13 seconds, since the unbounded rebuild was O(T) per call, O(T²) total. A LibriSpeech utterance's transcript is byte-identical before and after.
1 parent 16fda17 commit 2bf8895

1 file changed

Lines changed: 62 additions & 15 deletions

File tree

‎src/parakeet_capi.cpp‎

Lines changed: 62 additions & 15 deletions
Original file line numberDiff line numberDiff line change
@@ -66,9 +66,16 @@ struct parakeet_ctx {
6666
struct parakeet_stream {
6767
parakeet_ctx* ctx = nullptr; // borrowed (must outlive the stream)
6868
std::unique_ptr<pk::StreamingMel> mel; // incremental log-mel front end
69-
std::vector<float> mel_buf; // accumulated mel [n_mels, mel_T] feat-major
69+
// mel_buf holds ONLY the still-reachable window of mel history, feat-major
70+
// [n_mels, mel_T - mel_buf_origin]; column 0 is ABSOLUTE frame mel_buf_origin
71+
// (see append_mel_frames / feed_available's window()). Frames strictly
72+
// before mel_buffer_idx - pre_encode_cache_size() can never be read again
73+
// (mel_buffer_idx only advances -- see feed_available), so they are dropped
74+
// as they age out instead of being retained for the life of the stream.
75+
std::vector<float> mel_buf;
7076
int n_mels = 0;
71-
int mel_T = 0; // total mel frames accumulated so far
77+
int mel_T = 0; // total mel frames accumulated so far (absolute)
78+
int mel_buf_origin = 0; // absolute frame index of mel_buf's column 0
7279
std::unique_ptr<pk::StreamingSession> sess;
7380
int mel_buffer_idx = 0; // next un-fed mel frame (chunk schedule)
7481
bool first_chunk = true; // chunk 0 has no pre-encode overlap
@@ -77,25 +84,60 @@ struct parakeet_stream {
7784

7885
namespace {
7986
// Append `n_new` feat-major mel frames `[n_mels, n_new]` to the stream's
80-
// accumulated feat-major mel buffer `[n_mels, mel_T]`, growing mel_T. Both are
81-
// feat-major (out[m*T + t]); appending along the time axis requires a per-row
82-
// rebuild since the inner stride changes when T grows.
87+
// accumulated feat-major mel buffer, growing mel_T, and drop the prefix that
88+
// feed_available's window() can provably never read again.
89+
//
90+
// ROOT CAUSE of mudler/parakeet.cpp#63 (the streaming RSS leak): the mel buffer
91+
// used to retain the FULL stream history from mel_buffer_idx==0 forever, so
92+
// this function rebuilt a `[n_mels, T]` array from scratch on EVERY stream_feed
93+
// call with T monotonically growing for the life of the stream -- not just O(T)
94+
// CPU per call (the reason this rebuild exists at all, per the original
95+
// comment), but a continuous sequence of UNIQUELY, EVER-LARGER-sized heap
96+
// allocations. A general-purpose allocator can never satisfy a strictly-larger
97+
// request from a smaller freed block, so the freed (now too-small) blocks from
98+
// every earlier call accumulate as unusable, unreturned pages instead of being
99+
// recycled -- the process's live working set stays tiny (a few MB) but the
100+
// CUMULATIVE allocator debt grows with the stream's length, which is exactly
101+
// why the upstream report finds it reclaimed by neither stream_free nor a
102+
// fresh stream_begin: it is allocator-level fragmentation from the discarded
103+
// buffers' sizes, not stream-owned memory.
104+
//
105+
// feed_available's window(lo, hi) only ever reads lo >= mel_buffer_idx -
106+
// pre_cache, and mel_buffer_idx is monotonically non-decreasing (it only moves
107+
// forward by chunk_size). So at the START of any call here (mel_buffer_idx
108+
// unchanged since the previous feed_available pass), every mel frame before
109+
// mel_buffer_idx - pre_cache is provably dead and safe to drop; mel_buf then
110+
// stays bounded to O(pre_cache + chunk_size) -- a small constant -- instead of
111+
// O(stream length), for the life of the stream.
83112
void append_mel_frames(parakeet_stream* s, const std::vector<float>& frames, int n_new) {
84113
if (n_new <= 0) return;
85114
const int n_mels = s->n_mels;
86-
const int old_T = s->mel_T;
115+
const int old_T = s->mel_T; // absolute frame count so far
87116
const int new_T = old_T + n_new;
88-
std::vector<float> out((size_t)n_mels * new_T);
117+
118+
const int pre_cache = s->sess ? s->sess->pre_encode_cache_size() : 0;
119+
int keep_from = s->mel_buffer_idx - pre_cache;
120+
if (keep_from < s->mel_buf_origin) keep_from = s->mel_buf_origin; // never move backward
121+
if (keep_from > old_T) keep_from = old_T; // never past what exists
122+
123+
const int old_local_T = old_T - s->mel_buf_origin; // mel_buf's current column count
124+
const int drop_local = keep_from - s->mel_buf_origin; // columns to drop from the front
125+
const int kept = old_local_T - drop_local; // columns carried forward
126+
const int new_local_T = kept + n_new;
127+
128+
std::vector<float> out((size_t)n_mels * new_local_T);
89129
for (int m = 0; m < n_mels; ++m) {
90-
// copy existing [0, old_T)
91-
for (int t = 0; t < old_T; ++t)
92-
out[(size_t)m * new_T + t] = s->mel_buf[(size_t)m * old_T + t];
93-
// append new [old_T, new_T)
130+
// carry forward the still-reachable tail [drop_local, old_local_T)
131+
for (int t = 0; t < kept; ++t)
132+
out[(size_t)m * new_local_T + t] =
133+
s->mel_buf[(size_t)m * old_local_T + (drop_local + t)];
134+
// append the newly-arrived frames
94135
for (int t = 0; t < n_new; ++t)
95-
out[(size_t)m * new_T + (old_T + t)] = frames[(size_t)m * n_new + t];
136+
out[(size_t)m * new_local_T + (kept + t)] = frames[(size_t)m * n_new + t];
96137
}
97138
s->mel_buf.swap(out);
98139
s->mel_T = new_T;
140+
s->mel_buf_origin = keep_from;
99141
}
100142
} // namespace
101143

@@ -492,20 +534,25 @@ std::string feed_available(parakeet_stream* s, bool flush, int& eou_flag,
492534
const size_t ev0 = sess.events().size();
493535

494536
const int n_mels = s->n_mels;
495-
const int T = s->mel_T;
537+
const int T = s->mel_T; // absolute total frame count
496538
if (T <= 0) return std::string();
497-
const std::vector<float>& mel = s->mel_buf; // [n_mels, T] feat-major
539+
const int origin = s->mel_buf_origin; // absolute index of mel_buf's column 0
540+
const int local_T = T - origin; // mel_buf's actual (bounded) column count
541+
const std::vector<float>& mel = s->mel_buf; // [n_mels, local_T] feat-major, columns [origin, T)
498542

499543
const int chunk0 = sess.chunk_size_first();
500544
const int chunk_main = sess.chunk_size();
501545
const int pre_cache = sess.pre_encode_cache_size();
502546

547+
// lo/hi are ABSOLUTE frame indices (as before); translate into mel_buf's
548+
// own local (origin-relative) columns. append_mel_frames guarantees
549+
// lo >= mel_buffer_idx - pre_cache is always still resident.
503550
auto window = [&](int lo, int hi) {
504551
const int len = hi - lo;
505552
std::vector<float> w((size_t)n_mels * len);
506553
for (int m = 0; m < n_mels; ++m)
507554
for (int t = 0; t < len; ++t)
508-
w[(size_t)m * len + t] = mel[(size_t)m * T + (lo + t)];
555+
w[(size_t)m * len + t] = mel[(size_t)m * local_T + (lo - origin + t)];
509556
return w;
510557
};
511558

0 commit comments

Comments
 (0)