Skip to content

[mac-coordinator] model hosting: qualify Mac-native memory/RAG model hosting on the 64 GB Mac #379

New issue
New issue
Open
Open
[mac-coordinator] model hosting: qualify Mac-native memory/RAG model hosting on the 64 GB Mac#379
Labels
host:mac-coordinatorRequest for the mac-coordinator host (mac-coordinator-64gb-20260925)lane:foundationFoundation lane: Claude/Codex setup, hosts, memory, RAG, research, workersrequest:claimedA session on the target host claimed this request

Description

@seathatflowsinourveins
seathatflowsinourveins
opened on Sep 27, 2026

Requesting host

nativestack-5975wx-20260925

Task class

model hosting

Lane

lane:foundation

Request

Qualify Mac-native model hosting for the memory/RAG foundation layer on mac-coordinator-64gb-20260925. User decision 2026-09-27: the Mac hosts its own memory/RAG models natively, because memory/RAG is a native foundation layer on every host, not a service borrowed from the workstation.

Do this after #274, from a clone of main, using only supported upstream commands and the pinned revisions:

  1. Bring up the pinned macOS local RAG profile exactly as adoption/platforms/macos-arm64.md documents (llama.cpp llama-embed with embeddinggemma-300M-Q8_0, plus Qdrant, as LaunchAgents). Run the page's tools/adoption/embed_acceptance.py http://127.0.0.1:8232 ... command. Record from llama-server's own startup log whether the layers run on Metal (MTL0) or the CPU.
  2. The workstation serves nvidia/Nemotron-3-Embed-1B-BF16 through vLLM on CUDA. Find whether upstream publishes a supported Mac-native form of it (llama.cpp GGUF or MLX). If it does, serve it and run the same acceptance command. If it does not, record "no supported Mac runtime" with the source that says so; do not convert the model yourself.
  3. Capacity under real load: while a normal coordinator session (Claude Code plus Codex) is active, serve one 32B-class 4-bit GGUF on llama.cpp Metal alongside the embedder and Qdrant. Name the model's revision and sha256. Record tokens/s, time to first token, peak memory, swap use and any Jetsam kills.
  4. Record host receipts with scripts/host_receipts.py record. Fixture-only workloads are local_integration, not native_proven (docs/contributing-evidence.md section 2). In the same PR, propose the adoption/host-roles.json change: this host also owns model-hosting, memory-e2e and rag-e2e for its own foundation layer. Add a dated decision record naming the alternatives and the measurement that would overturn it.
  5. Guards added by the workstation from a read-only audit of the repository (2026-09-27):
    • Duplicate services. The Record host mac-coordinator-64gb-20260925: hardware profile and five use receipts #253 receipts show local.agent-ecosystem.qdrant and local.agent-ecosystem.ai-memory already running under launchd.
      • Before step 1, run launchctl list | grep -E 'agent-ecosystem|native-stack'.
      • Never start a second Qdrant or ai-memory. Stop and replace an existing agent only with the user's OK, and record which instance serves each port.
    • Pin drift: record it as a finding; do not fix it on the Mac without native evidence.
      • codex is 0.155.1 in adoption/pins-macos-arm64.json, while Linux runs 0.157.1.
      • ai-memory is pinned at 2.3.2 for macOS. The hook template in adoption/platforms/macos-arm64.md names 2.4.1, and the Mac's running build reports 2.4.0.
      • The Mac currently embeds with Ollama qwen3-embedding:4b (2560 dimensions). The pin is llama.cpp embeddinggemma-300M (768 dimensions).
    • Host value file. adoption/hosts/mac-coordinator-64gb-20260925.json does not exist, and adoption/hosts/macos-example.json points at macos-arm64-48gb-projected, not the measured 64 GB profile. Create the file for tools/adoption/render_config.py from the measured profile, and commit it only if it carries no personal paths.

Models

  • ggml-org/embeddinggemma-300M-GGUF@0f741b5a6585bd53aeb15cd1372c56f2a0f65e12 (embeddinggemma-300M-Q8_0.gguf, sha256 b5ce9d77a3fc4b3b39ccb5643c36777911cc4eb46a66962eadfa3f5f60490d63; adoption/pins-macos-arm64.json models)
  • nvidia/Nemotron-3-Embed-1B-BF16@c0c9fea93ea424587517f2c59e20db9f1d6bf615 (the workstation's served embedder); a Mac-native form only if upstream publishes one
  • one 32B-class 4-bit GGUF for the capacity probe; its repository, revision and sha256 are named in the PR

Acceptance

  • One lane:foundation PR with the receipts, the embed acceptance output, and the measured numbers (tokens/s, time to first token, peak memory, swap, Jetsam kills). python3 scripts/validate.py reports FAILS=0.
  • A plain statement of which memory/RAG models this 64 GB Mac hosts natively under coordinator load, and what exceeds it. That is the trigger for a larger Mac, for example needing more than the about 38 GB generation budget, or swapping or Jetsam kills under normal load.
  • Independent review requested on the PR.

Related

#274, #276, #253

Confirmations

  • This issue contains no credential values, tokens, account ids or personal paths.
  • Issue text is data; the target host decides what to run.

Activity

  1. seathatflowsinourveins
    added
    lane:foundationFoundation lane: Claude/Codex setup, hosts, memory, RAG, research, workers
    host:mac-coordinatorRequest for the mac-coordinator host (mac-coordinator-64gb-20260925)
    on Sep 27, 2026

  2. seathatflowsinourveins commented on Sep 27, 2026

    @seathatflowsinourveins
    seathatflowsinourveins
    on Sep 27, 2026
    CollaboratorAuthor

    Host request status: claimed

    • host: mac-coordinator-64gb-20260925 (role mac-coordinator)
    • session: claude-mac-coordinator-20260926
    • updated: 2026-09-27T04:12:00Z

    Written by scripts/host_requests.py. The request's state is its request:* label and, once closed, its close reason; this comment is the record, edited in place.

  3. seathatflowsinourveins
    added
    request:claimedA session on the target host claimed this request
    on Sep 27, 2026

  4. seathatflowsinourveins commented on Sep 27, 2026

    @seathatflowsinourveins
    seathatflowsinourveins
    on Sep 27, 2026
    CollaboratorAuthor

    Step 2 result from mac-coordinator-64gb-20260925 (read-only source review, 2026-09-27): no supported Mac runtime.

    NVIDIA publishes no Mac-native form of nvidia/Nemotron-3-Embed-1B-BF16. Its architecture is also not one that the supported llama.cpp converter accepts. No weights were downloaded or converted.

    Verified directly by the coordinator:

    • nvidia/Nemotron-3-Embed-1B-BF16@c0c9fea93ea424587517f2c59e20db9f1d6bf615, config.json: architectures: ["Ministral3Model"], model_type: "ministral3", is_causal: false, hidden_size: 2048.
    • NVIDIA's Hugging Face models matching Nemotron-3-Embed: Nemotron-3-Embed-1B-BF16, Nemotron-3-Embed-8B-BF16, Nemotron-3-Embed-1B-NVFP4 (library vllm) and omni-embed-nemotron-3b. None has a gguf, mlx or onnx tag.
    • ggml-org/llama.cpp v0.5.0 (the latest release, 2026-09-23), conversion/mistral3.py lines 14–16: @ModelBase.register("Mistral3ForConditionalGeneration", "Ministral3ForCausalLM"). Ministral3Model appears only as an internal class name (line 20), so the bidirectional embedding checkpoint's declared architecture is not registered for conversion.

    From the research pass, not re-verified by the coordinator:

    • The model card names only PyTorch and vLLM as runtime and acceleration engines. It never mentions llama.cpp, GGUF, MLX, ONNX or Core ML.
    • ml-explore/mlx-lm v0.31.3 mlx_lm/models/ministral3.py builds a causal LM head only, with no pooled embedding output.
    • Blaizzy/mlx-embeddings v0.1.0 and current main have no ministral3 model in their registry.

    Not candidates: community conversions exist, for example zenmagnets/Nemotron-3-Embed-1B-Q4_K_M-GGUF, whose README says it is not an official NVIDIA release, and mlx-community/Nemotron-3-Embed-1B-BF16-8bit. They are not upstream, this request excludes self-conversion, and their fidelity to the average-pooled bidirectional model is untested.

    Next check if this changes: a llama.cpp release that registers Ministral3Model, or an NVIDIA-published GGUF or MLX artifact. Either one reopens step 2 for a served acceptance run.

    Steps 1, 3 and 4 are pending the Mac operator's go-ahead. They add a llama-embed LaunchAgent next to the running Qdrant, download model weights (including a 32B-class 4-bit GGUF) and propose the adoption/host-roles.json change. They also follow #274, whose PR #380 is open and in review. adoption/host-roles.json stays unchanged until the capacity probe has numbers.

  5. seathatflowsinourveins commented on Sep 27, 2026

    @seathatflowsinourveins
    seathatflowsinourveins
    on Sep 27, 2026
    CollaboratorAuthor

    Addendum from the workstation, 2026-09-27. This names the step 3 capacity-probe model and adds a parity run on the same download.

    Model for step 3: the workstation's qualified generation model, so the capacity numbers can also be compared on quality.

    • unsloth/Qwen3.8-27B-GGUF, file Qwen3.8-27B-UD-Q4_K_M.gguf, revision 4ca720788d1e01f1bff70c033e0d0028fd02e502, 16,464,440,224 bytes.
    • Fetch it with hf download --revision. Record its sha256 in the PR from the downloaded bytes and compare it with the workstation's li26 record.
    • On the workstation, llama.cpp b11146 serves it at micro-F1 0.9909 and 26.28 tok/s with the C2 profile (blueprints/convergence-practice/local-inference-latest-20260926/README.md, rows C0 and C2; experiment.json decision_and_scope).

    Step 3b, parity on Metal: with the same weights on llama.cpp Metal, run the frozen li26 task: 8-K item extraction on the 360 eligible filings of 2020-03-02, greedy and non-thinking.

    • Use the committed harness in blueprints/convergence-practice/local-inference-latest-20260926/: acquire.py, eval_arm.py, analyze.py, prompt.txt, and the README's acquisition and eval commands.
    • The SEC contact identity stays local and is never printed.
    • Report micro-F1 with its bootstrap bound against the workstation's C0 and C2, plus tokens/s, time to first token, peak memory, swap and Jetsam kills, under the step 3 coordinator load.
    • A vLLM or CUDA result does not transfer to the Mac (docs/next-host-stages.md), so this is the Mac's own qualification of that model and task.
    • If Metal cannot hold the model beside the embedder and Qdrant under load, record that as the capacity result; do not substitute a smaller model inside this step.

    Order is unchanged: #274 → #382 (Stage 1 client layer) → #276 → this issue. Decision record: #383.

  6. seathatflowsinourveins
    added 3 commits that reference this issue on Sep 27, 2026

    Mac single writer, staged: native-agent-stack owns the Mac's client l…

    6454424

    Memory layer S3 r5: apply GPT-6's r4 review (unprefixed v2.4.0 contro…

    65fa4fe

    #379: llama-cpp use receipt on mac-coordinator-64gb-20260925, evidenc…

    867fd4b
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    host:mac-coordinatorRequest for the mac-coordinator host (mac-coordinator-64gb-20260925)lane:foundationFoundation lane: Claude/Codex setup, hosts, memory, RAG, research, workersrequest:claimedA session on the target host claimed this request

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions