Linux x86-64 native engine with LoRA, prerouter and memory budget #165

Area

New platform support

What problem are you trying to solve?

The published Edge0 8B and 35B models have no native engine on Linux today: the Python MLX runtime targets Apple Silicon, and the Windows engine uses Windows memory APIs. In #18 a user asked whether Linux support is planned.

Proposed solution

A Linux x86-64 native engine that follows the existing pinned llama.cpp and patch-band layout, with the implementation in the linked PR: CPU/AVX2 by default, isolated common + linux patch replay, POSIX RSS accounting and page reclamation, bounded expert-page advice and an optional expert cache. It reuses the existing Windows model and LoRA converters and serves /v1/chat/completions on loopback.

Measured on an Intel i9-9900K (AVX2, 62 GiB RAM, Ubuntu 24.04 host) in a Debian bookworm / GCC 12.2 container:

The 8B figures are three-run medians without an Edge0 memory budget; the 35B figures are one observation per workload with a 12 GiB Linux budget and no vanilla budget. These are aggregate server decode rates measured on a shared machine under load, so they are provisional; a quiet-machine comparison with vanilla is still pending.

Both tiers returned chat answers with LoRA, prerouter and page-advice activity confirmed in the logs. The 35B peaked at 11,894.1 MiB RSS with a 12,288 MiB budget and at 3,890.4 MiB with a 4,096 MiB budget. Reclamation is cooperative: transient overruns are possible and failures are logged; it is not a hard RSS ceiling. The optional expert cache has unit coverage and was off for these measurements.

The native and sanitizer test suites pass, and the 8B four-mode smoke test kept greedy tokens and GGUF bytes identical. CUDA and Vulkan build flags exist, but only the Vulkan configuration step was verified; no GPU build or inference is claimed.

Alternatives considered

Pristine llama.cpp is the measured baseline but does not activate the Edge0 LoRA, prerouter and budget integration. A separate Linux band and serving directory keep platform-specific behavior scoped, without changes to the existing platforms.

The change adds about 4,400 lines, mostly new files under linux/ plus two Linux patches. Please let me know whether this scope and layout fit the project, or whether you would prefer it split into smaller PRs.

AI assistance: OpenAI Codex and Claude Code were used to implement, test and document this work.