…ion masks (#49)

In a heterogeneous-length batch on a chunked_limited / local-attention
model (e.g. nemotron-3.5-asr-streaming-0.6b), any clip whose padding tail
exceeded the attention left context (~4.5 s of length spread at 8x
subsampling) decoded to an EMPTY transcript. Mechanism: a deeply padded
query row has its ENTIRE additive mask row at -inf, softmax over all--inf
yields NaN, the post-softmax query-row mask keeps it (NaN*0 == NaN), and
the NaN key columns then poison every softmax row of the next layer --
24 layers later the whole item is NaN. Full attention was unaffected
(padded query rows can still see valid keys), which is why only
limited-context models in batches with enough length spread were hit; the
batch JSON entry points the LocalAI backend calls were silently affected.

Leave fully-masked padded query rows unmasked instead: their finite
garbage output is zeroed by the existing post-softmax query mask (output
reduces to linear_out.bias, exactly as before), and valid rows' masks are
untouched. Applied to every band/window mask builder (single + batched,
full/chunked-limited + banded local + chunked kernel).

Verified on nemotron-3.5-asr-streaming-0.6b q8_0 (att_context [56,3]): in
a {16.4 s, 7.4 s} batch the short item's encoder output was 100% NaN
before the fix and now byte-matches its solo decode; transcripts match
solo at every tested T_max spread (8.4-16.4 s). Also verified on
parakeet_realtime_eou_120m-v1 f16 (att_context [70,1]): pre-fix the 7.4 s
fixture decoded empty in any batch with T_max >= 14.4 s, post-fix it
byte-matches solo at every spread. Cache-aware streaming and single-clip
offline transcripts are byte-identical between pre-fix and post-fix
builds (those paths never form padded batches), so the fix changes
nothing outside the broken case.

Assisted-by: Claude:claude-fable-5 [Claude Code]

* build(arm64): cross-compile parakeet-cli for the Raspberry Pi 5

Static, self-contained aarch64 build via docker buildx + qemu, mirroring the
privacy-filter.cpp approach. Base ubuntu:24.04 matches the Pi 5's glibc 2.39 ABI
so the binary drops straight on; it targets the Cortex-A76
(armv8.2-a+dotprod+fp16) and links only the system libc/libstdc++/libgomp (ggml
and libparakeet are static).

  docker buildx build --platform linux/arm64 -f docker/Dockerfile.arm64 \
      --target export --output type=local,dest=build/arm64 .

The build context drops .git (.dockerignore), which dangles the ggml submodule's
gitlink and breaks scripts/apply_ggml_patches.sh (it uses `git apply`). Re-init a
throwaway git repo in the submodule tree before configure so the in-tree ggml
patches (the sgemm broadcast-fold, etc.) apply -- worth ~9% at 4 threads on the Pi.

Co-Authored-By: Claude Opus 4.8 (1M context) <[email protected]>

* build(appimage): package the static parakeet-cli builds as AppImages

Binary-only AppImages (no model inside) for aarch64 and x86_64.
packaging/appimage/make-appimage.sh is the single packaging path: it
assembles the AppDir (parakeet-cli plus the only two non-libc dynamic
deps, libstdc++/libgomp, resolved from the running system), fetches
appimagetool and the type2 runtime, and packs
parakeet-cli-<version>-<arch>.AppImage. glibc stays system-provided, so
the AppImage carries the same glibc floor as the raw binary.

docker/Dockerfile.arm64 becomes docker/Dockerfile.static: same static
build, but the arch flags now key off TARGETARCH (arm64 keeps the Pi 5
Cortex-A76 default; amd64 pins a portable Haswell+ AVX2 baseline since
GGML_NATIVE=OFF) and a new appimage/export-appimage stage pair packages
both arches in one buildx invocation. The downloaded appimagetool gets
its three AppImage magic bytes zeroed before use: qemu-user's binfmt
mask requires the ELF e_ident pad bytes to be zero, so the aarch64 tool
otherwise dies with "Exec format error" under emulation.

release.yml grows a Package/Smoke/Upload AppImage step trio on the
linux cpu matrix entries (native runners, same script), so tagged
releases ship one grab-and-run file per arch next to the tarballs.
vulkan/cuda stay tarball-only -- their binaries need system GPU stacks
an AppImage can't sensibly bundle.

Verified: x86_64 AppImage in a clean ubuntu:24.04 container and the
aarch64 one on a Pi 5 (direct FUSE exec) both print --version and
transcribe the speech.wav fixture byte-identically to the known-good
text.

Co-Authored-By: Claude Fable 5 <[email protected]>

* bench(rpi5): benchmark all published model sizes on a fan-cooled Pi 5

Pi 5 Model B Rev 1.1 (8 GB, Active Cooler, Raspberry Pi OS trixie),
static aarch64 build from docker/Dockerfile.static, governor pinned to
performance. scripts/bench_rpi5.sh drives parakeet-cli bench over a
model x threads{1,2,4} x clip{7.4s,67s} matrix with a 2s
temp/clock/fan/throttle sampler beside every run; benchmarks/rpi5.md
has the tables, benchmarks/results/rpi5/ the raw JSON + telemetry.

Headlines: tdt_ctc-110m-q8_0 hits 18x real time at 4 threads;
tdt-0.6b-v3-q8_0 and nemotron q8_0 hold 2.8-4.6x at 4 threads and stay
>= 1x single-threaded on the long clip at ~1.0 GB peak RSS. The fan
answers the question it was bought for: all 24 cells ran the full
2.4 GHz with throttled=0x0, peaking at 77.1 C against the 85 C soft
limit. All four models transcribe the fixture correctly on-device
(nemotron with no LID-tag leakage).

Co-Authored-By: Claude Fable 5 <[email protected]>

---------

Co-authored-by: Claude Opus 4.8 (1M context) <[email protected]>

* feat(tdt): add opt-in N-best beam decoding

Implement NeMo-compatible default TDT beam search, expose ranked hypotheses through Model, C API, and CLI surfaces, and add CPU/CUDA plus closed-loop NeMo parity coverage.

Assisted-by: Codex:gpt-5 [Codex]

* fix(tdt): harden N-best edge cases

Validate requests before inference, reject malformed duration tables and non-progressing zero-duration expansions, and broaden real-model API coverage.

[skip ci]

Assisted-by: Codex:gpt-5 [Codex]

* fix(tdt): score legal blank duration fallback

Select the highest-scoring positive duration when zero is the only top-k blank duration, and cover the beam-1 fallback independently of a model.

Assisted-by: Codex:gpt-5 [Codex]

* fix(test): include invalid_argument definition

Keep the TDT beam test portable across compilers instead of relying on a transitive standard-library include.

Assisted-by: Codex:gpt-5 [Codex]

---------

Co-authored-by: inlanger <[email protected]>

* workflows: adds a cuda12 variant to docker workflow

* workflows: release a cude12/linux variant for older GPUs like Volta

* workflows: remove 12.1 from cuda12/arm

* Add parakeet_capi_transcribe_pcm_logits: expose CTC log-probs

The C-API only ever returned decoded text/timestamps/JSON, even though
CTCDecoder::forward already computes a log-softmaxed [T, vocab+1] matrix
internally. External decoder stacks that do their own LM fusion (e.g.
pyctcdecode + KenLM + hotwords) need that matrix directly instead of
this library's own greedy/beam decode.

Adds Model::transcribe_pcm_ctc_logits (mirrors transcribe_pcm through
mel + encoder, including the long-audio tiling path, then runs the CTC
head only, skipping decode) and the matching C-API entry point +
parakeet_capi_free_logits, following the existing validate/try-catch/
last_error conventions used by every other capi_* function. Bumps the
ABI version to 6.

Verified against the real parakeet-ctc-1.1b checkpoint (tests/test_capi_ctc_logits.cpp,
gated on PARAKEET_TEST_GGUF_CTC): argmax-greedy decode over the exposed
matrix reproduces transcribe_pcm(..., kCTC)'s own greedy text
byte-for-byte, and matches the fixture's known NeMo reference.

* Add no-CTC-head error path coverage to test_capi_ctc_logits

Verifies the C-API boundary contract directly: a model with no CTC head
(a pure RNNT/TDT streaming model) makes parakeet_capi_transcribe_pcm_logits
fail cleanly (nonzero return, *out_logits left NULL, last_error set)
rather than letting ctc_head_tensor's std::runtime_error cross the C
boundary. Gated on PARAKEET_TEST_GGUF_NO_CTC, independent of the existing
self-consistency block.

* Address review: validation order, output zeroing, C-API test coverage

- parakeet_capi_transcribe_pcm_logits: check ctx before the output
  pointers (so an invalid out-param can still set ctx->last_error), and
  zero *out_T/*out_vocab_plus_1 alongside *out_logits up front so every
  failure path leaves defined state, matching the header's documented
  contract.
- tests/test_capi_ctc_logits.cpp block 1: call the actual
  parakeet_capi_load / parakeet_capi_transcribe_pcm_logits /
  parakeet_capi_free_logits C-API instead of Model::transcribe_pcm_ctc_logits
  directly, so the test exercises the C boundary and malloc/free contract
  on the success path too (pk::Model is now used only for the reference
  text and tokenizer/blank_id).

* Address review: halve test peak RAM, fix misleading prompt comment

- tests/test_capi_ctc_logits.cpp block 1: cache blank_id/tokenizer_pieces
  and release the pk::Model (model.reset()) before loading a second full
  model via parakeet_capi_load — the two were resident simultaneously,
  nearly doubling peak RAM for a 1.1B checkpoint and risking CI OOM.
- src/model.cpp: transcribe_16k_ctc_logits's prompt-conditioning comment
  claimed CTC-only models are never prompt-conditioned "in practice",
  which overclaims a current-catalog observation as if it were a property
  the function depends on. Reworded to match transcribe_16k's existing,
  accurate wording (no-op when prompt.present == false, full stop).