Summary

Fixes #50 — embedding-fastembed on Linux/Windows force-enabled ort-load-dynamic, which silently reproduced as an infinite futex deadlock instead of a clear error.

Two coupled defects, both fixed:

1. Cargo feature interaction (the trigger)

ort-load-dynamic forwards to ort-sys/disable-linking, which makes ort-sys's build script early-return and skip the static-archive download step entirely. So ort-download-binaries-rustls-tls was a silent no-op, and the resulting binary expected libonnxruntime.so to be supplied externally at runtime — which llm-kernel never ships.

Fix: Drop ort-load-dynamic from the Linux/Windows fastembed target dep. The default build now statically links ONNX Runtime and produces self-contained binaries (verified: libloading drops out of Cargo.lock, cargo:rustc-link-lib=static=onnxruntime is emitted).

2. LazyFastembedProvider panic safety (the hang)

A panic during FastembedProvider::new() (e.g. ort init panicking on a missing dylib — exactly what load-dynamic produced) unwound across the Mutex/Condvar boundary without transitioning ModelState to Failed or notifying, wedging any peer thread parked in the Loading branch forever. Confirmed in production via strace / /proc/PID/wchan (futex_wait_queue, 0% CPU, no syscalls).

Fix: Wrap the load in std::panic::catch_unwind(AssertUnwindSafe(..)); convert a panic into ModelState::Failed("model init panicked: ..") + cvar.notify_all(). Guards against any future ort/fastembed init failure mode, not just the dynamic-linking one.

Why no embedding-fastembed-dynamic opt-in feature?

Investigated and rejected. Cargo feature unification merges fastembed features into a union across llm-kernel features — so any embedding-fastembed-dynamic feature (which must imply embedding-fastembed) would re-enable both ort-download-binaries AND ort-load-dynamic together whenever both were selected, silently reproducing #50 with no compile-time error. Verified empirically via cargo metadata + cargo check -vv showing CARGO_FEATURE_DISABLE_LINKING=1 + CARGO_FEATURE_DOWNLOAD_BINARIES=1 set simultaneously.

Cargo's mutually-exclusive-features mechanism (implied-by) is unstable and can't guard this. Deployments that truly need dynamic loading should depend on fastembed directly in their own crate and enable ort-load-dynamic there, outside llm-kernel's feature graph.

Audit

• Concurrency: PASS — catch_unwind placement, all three outcomes (Ok(Ok)/Ok(Err)/Err(panic)) transition state + notify_all exactly once, Inner mutex never held across the load (Arc::clone + drop(guard)), AssertUnwindSafe soundness verified (value-captured inputs, no torn state leaks), TOCTOU-free waiter wakeup.
• Build/Cargo: PASS (after fix) — initial audit caught the feature-unification defect, fixed by removing the dynamic feature.
• Tests: PASS — 543 passed, 0 failed, 6 ignored (was 540; +3 new panic-safety regression tests).

Tests added

• panic_during_load_transitions_to_failed — a panicking loader → state() == Failed("model init panicked: ..").
• failure_during_load_transitions_to_failed — a failing loader → Failed.
• concurrent_waiter_released_when_loader_panics — AC4: a waiter parked in Loading is released with Err (not hung) when the owning loader panics; verified with a real spawned thread + barrier, no load_timeout_secs wait.

Tests use an injected loader (new_with_loader) so no real ONNX weights are required.

Behavior change

Consumers that relied on the (broken) dynamic-linking default must now statically link (the new default) — which is what they actually wanted. The dynamic path is documented in CHANGELOG for the rare deployment that needs it.

Verification

cargo test --all-features          # 543 passed, 0 failed, 6 ignored
cargo clippy --all-features -- -D warnings   # clean
cargo fmt --all -- --check         # clean
cargo build --features embedding-fastembed    # static link confirmed