feat(runtime): model metadata L3 cache + L4 Ollama probe (PR-2/3)#3134
Merged
Conversation
|
You have reached your Codex usage limits for code reviews. You can see your limits in the Codex usage dashboard. |
houko
force-pushed
the
feat/model-metadata-lookup-m1
branch
from
April 25, 2026 13:43
4609610 to
17e1b50
Compare
2 tasks
Builds on PR-1 (#3133) by lighting up the persisted-cache and Ollama-runtime-probe layers in the lookup pipeline. Still passive — no kernel call site invokes `resolve_model_metadata` yet, so runtime behaviour is unchanged. PR-3 will wire the kernel and retire the uniform 200K default. `resolve_model_metadata` is now `async` (was sync in PR-1) — required for the spawn_blocking-wrapped fs IO in L3 and the reqwest-driven HTTP probe in L4. The signature also grows a `home_dir: &Path` arg so the cache file path is dependency-injected (kernel will pass `config.home_dir`). Layer 3: persisted cache - `<home_dir>/cache/model_metadata.json` with `{ version, entries }` schema, atomic-rename writes, schema-version mismatch handled by silent reset (warns + treats as empty cache). - Cache key: `provider|base_url|model` triple — same model id under different providers / endpoints stays separated. - 24h TTL; entries past their fetched_at + ttl fall through to L4. - All file IO goes through `tokio::task::spawn_blocking` so the runtime never stalls on slow disks. Layer 4: Ollama `/api/show` probe - 3-second hard timeout (mirrors hermes-agent's `_PROBE_TIMEOUT`). - `parse_ollama_show` prefers the `parameters` block's `num_ctx` (effective server cap) over the nominal `model_info.*.context_length` (theoretical model max). Zero values rejected — caching `0` would break downstream budget math. - `looks_like_ollama` matches the literal `ollama` provider tag and any base_url containing `:11434` (canonical Ollama port). - HTTP client is a process-wide `OnceLock<reqwest::Client>` built from `librefang_http::proxied_client` so corporate-proxy environments work transparently. - L4 dispatcher only routes to Ollama in this PR; PR-2.5 will add Anthropic `/v1/models` + generic OpenAI-compat `/v1/models/{id}`. Other changes: - `MetadataRequest` grows `api_key: Option<&'a str>` (will be needed for the Anthropic probe in PR-2.5). - All 13 PR-1 unit tests refactored to `#[tokio::test]` + a tempdir-backed `home_dir`. - `MetadataSource::PersistedCache` and `RuntimeProbe` are no longer placeholders. Tests (9 new, 23 total in `model_metadata::tests`): - `layer_3_cache_round_trip` — write entry → resolve picks it up at L3, source = PersistedCache. - `layer_3_stale_cache_falls_through` — entry fetched 25h ago is rejected; falls to L5 hardcoded. - `layer_3_corrupted_file_does_not_panic` — garbage cache file logs a warn and degrades to empty cache, not panic. - `parse_ollama_show_prefers_num_ctx` — effective cap wins over nominal `*.context_length`. - `parse_ollama_show_falls_back_to_model_info` — when `num_ctx` is absent, first `*.context_length` is used. - `parse_ollama_show_returns_none_on_missing_fields` — empty / no recognised fields → None (not a misleading 0). - `parse_ollama_show_rejects_zero` — `num_ctx 0` and `*.context_length: 0` both rejected. - `ollama_detection_provider_or_port` — `looks_like_ollama` matches both the literal tag and the canonical port. - `cache_key_separates_endpoint_namespaces` — same model on different hosts gets distinct cache keys. Verification: - cargo check --workspace --lib: clean - cargo test -p librefang-runtime --lib model_metadata: 23 ok - cargo clippy -p librefang-runtime --all-targets -- -D warnings: clean - Live integration deferred to PR-3 (cargo build is sandbox-blocked in this environment). Stack: this PR's branch is `feat/model-metadata-lookup-m2` based on PR-1 (`feat/model-metadata-lookup-m1`, #3133). When #3133 lands, this PR's diff against main collapses to PR-2's increments. PR-2.5 (next) will add Anthropic + OpenAI-compat L4 probes. PR-3 will wire `resolve_model_metadata` into kernel and retire the 200K default.
houko
force-pushed
the
feat/model-metadata-lookup-m2
branch
from
April 25, 2026 14:10
cb7985f to
d3342e8
Compare
houko
added a commit
that referenced
this pull request
Apr 25, 2026
…-2.5/3)
Extends the L4 runtime probe (introduced in PR-2 with the Ollama
branch only) to cover two more endpoint shapes:
- Anthropic `GET /v1/models/{model}` — uses the documented
`x-api-key` + `anthropic-version` headers; pulls `context_window`
from the top level. Requires `MetadataRequest.api_key` (added in
PR-2 and unused until now).
- Generic OpenAI-compat `GET /v1/models/{model}` — covers vLLM, LM
Studio, llama.cpp server, LiteLLM-style proxies. The per-model
schema isn't standardised, so the parser tries common keys in
priority order:
1. `max_model_len` (vLLM canonical)
2. `context_length` (LM Studio, llama.cpp, GGUF metadata)
3. `context_window` (Anthropic-flavoured proxies)
4. `max_input_tokens` (LiteLLM router)
5. `max_tokens` (legacy fallback)
Zero values rejected — caching `0` would break downstream
ContextBudget math.
`probe_runtime` dispatcher now routes:
- Ollama (literal tag or `:11434` port) → `probe_ollama`
- Provider == "anthropic" with api_key → `probe_anthropic`
- Otherwise, when caller provided `base_url` → `probe_openai_compat`
- Else → None (falls through to L5)
All probes share `PROBE_TIMEOUT_SECS = 3` and the process-wide
`OnceLock<reqwest::Client>` from PR-2. Successful probes write
back to L3 cache (already wired in PR-2).
Tests (10 new, 33 total in `model_metadata::tests`):
- parse_anthropic_model: extracts context_window, missing → None,
zero → None, wrong type → None.
- parse_openai_model: priority order (5 separate cases for each
recognised key), unrecognised payload → None, zero rejected.
- HTTP probe functions don't have unit tests (no wiremock dev-dep
per CLAUDE.md "禁止擅自加依赖"); live integration deferred to
PR-3 once `resolve_model_metadata` is wired into the kernel.
Verification:
- cargo test -p librefang-runtime --lib model_metadata: 33 ok
- cargo clippy -p librefang-runtime --all-targets -- -D warnings: clean
Module doc table updated to mark L4 fully live (no longer
"Ollama only").
Stack: this PR's branch is `feat/model-metadata-lookup-m2-5` based
on PR-2 (`feat/model-metadata-lookup-m2`, #3134). When #3134 lands,
this PR's diff against main collapses to PR-2.5's increments.
PR-3 (next) wires `resolve_model_metadata` into kernel and retires
the 200K default in `agent_loop.rs:1285`.
houko
added a commit
that referenced
this pull request
Apr 25, 2026
…-2.5/3) (#3140) Extends the L4 runtime probe (introduced in PR-2 with the Ollama branch only) to cover two more endpoint shapes: - Anthropic `GET /v1/models/{model}` — uses the documented `x-api-key` + `anthropic-version` headers; pulls `context_window` from the top level. Requires `MetadataRequest.api_key` (added in PR-2 and unused until now). - Generic OpenAI-compat `GET /v1/models/{model}` — covers vLLM, LM Studio, llama.cpp server, LiteLLM-style proxies. The per-model schema isn't standardised, so the parser tries common keys in priority order: 1. `max_model_len` (vLLM canonical) 2. `context_length` (LM Studio, llama.cpp, GGUF metadata) 3. `context_window` (Anthropic-flavoured proxies) 4. `max_input_tokens` (LiteLLM router) 5. `max_tokens` (legacy fallback) Zero values rejected — caching `0` would break downstream ContextBudget math. `probe_runtime` dispatcher now routes: - Ollama (literal tag or `:11434` port) → `probe_ollama` - Provider == "anthropic" with api_key → `probe_anthropic` - Otherwise, when caller provided `base_url` → `probe_openai_compat` - Else → None (falls through to L5) All probes share `PROBE_TIMEOUT_SECS = 3` and the process-wide `OnceLock<reqwest::Client>` from PR-2. Successful probes write back to L3 cache (already wired in PR-2). Tests (10 new, 33 total in `model_metadata::tests`): - parse_anthropic_model: extracts context_window, missing → None, zero → None, wrong type → None. - parse_openai_model: priority order (5 separate cases for each recognised key), unrecognised payload → None, zero rejected. - HTTP probe functions don't have unit tests (no wiremock dev-dep per CLAUDE.md "禁止擅自加依赖"); live integration deferred to PR-3 once `resolve_model_metadata` is wired into the kernel. Verification: - cargo test -p librefang-runtime --lib model_metadata: 33 ok - cargo clippy -p librefang-runtime --all-targets -- -D warnings: clean Module doc table updated to mark L4 fully live (no longer "Ollama only"). Stack: this PR's branch is `feat/model-metadata-lookup-m2-5` based on PR-2 (`feat/model-metadata-lookup-m2`, #3134). When #3134 lands, this PR's diff against main collapses to PR-2.5's increments. PR-3 (next) wires `resolve_model_metadata` into kernel and retires the 200K default in `agent_loop.rs:1285`.
houko
added a commit
that referenced
this pull request
Apr 25, 2026
Emergency fix — main is red on Quality + Test/macOS. The fmt residue came in via my rebase resolutions of #3134 / #3140 where I focused on conflict markers and didn't run cargo fmt locally before push. Both PRs were squash-merged into main, carrying the un-fmt'd lines along. Pure formatting, no behaviour change.
Merged
6 tasks
houko
added a commit
that referenced
this pull request
Apr 26, 2026
… / providers extras Final batch of catch-up docs covering the remaining Apr 22-25 user-facing features: - agent/hooks: transform_tool_result hook contract (#3003) - agent/templates: tool_search / tool_load lazy loading + lazy_tools opt-out (#3047) - integrations/mcp-a2a: TaintRuleId enum + per-tool skip_rules policy (#2999) - configuration/features: compaction summaries in user's conversation language (#3007) - configuration/providers/local: embedding auto-detection priority list (#3099) + Ollama capability detection (Modality / metadata pipeline, #3074 / #3133 / #3134 / #3140) - configuration/providers/management: custom image-gen provider via generic OpenAI-compat driver (#2998) + Moonshot file-upload helper via /v1/files (#2966) All sections mirrored to docs/src/app/zh/.
houko
added a commit
that referenced
this pull request
Apr 26, 2026
…pi / observability (#3189) * docs(providers): add Novita / SearXHG / Bedrock entries * docs(config): document [parallel_tools] section + max_history_messages * docs(scheduler): cron multi-delivery + pre_script + silent_marker * docs: live context.md, chat attachments, CLI slash commands Salvages three of the six items from the (quota-killed) misc-features agent run. Each is a focused append, no rewrite of existing content. - agent/templates: live Live Context section explains the per-turn re-read of `context.md` (#3115), the .identity/ vs root path lookup, the 32 KB cap with stale-cache fallback, and the cache_context opt-out. - integrations/api/agents: attachment-resolution table (image / PDF / text+code) covering the supported MIME / extension list and the 200 KB truncation, plus a note about adjacent-text coalescing for small chat-tuned local models (#3094). - integrations/cli/commands: 'Slash Commands in Chat' section listing the registry-backed CLI commands (/model /status /help /clear /kill /exit) and pointing at channel-side commands for the broader set (#3122 #3123). * docs: tool invoke API, trajectory export, doctor audit, CLI default-pick, SDK codegen Picks up the items the second misc-features agent didn't get to before hitting the org quota. - integrations/api/system: POST /api/tools/{name}/invoke (#3025) — fail- closed allowlist, agent_id requirement for approval-gated tools. - integrations/api/agents: GET /sessions/{id}/trajectory (#3097) — json/jsonl formats, redactor pipeline, audit use case. - operations/troubleshooting: 'Audit checks' subsection enumerating the three registered AuditChecks (#3082) — VaultKey, ApiListenAddr, ConfigTomlSchema — with what each asserts and how to extend. - configuration/providers/tools: 'CLI logins as first-class default providers' section (#3061) — detection priority claude-code > codex- cli > gemini-cli > qwen-code, override via [default_model] in config.toml. - integrations/sdk: 'Auto-generated from openapi.json' section (#3046) — scripts/codegen-sdks.py, pre-commit hook, openai-tag skip rule. * docs(zh): mirror providers / config-core / cron / features-index sections * docs(zh): mirror agent / api / cli / providers-tools / troubleshooting / sdk additions Round 2 of the zh translation pass — picks up the seven mid-priority files that the first commit didn't cover. All sections mirror the EN content from #3189 but read naturally as Chinese (terms like agent / session / fallback / NoEntry left in English where they're API surface, prose translated to 中文). - agent/templates: Live Context section, .identity/ vs root fallback, cache_context = true opt-out. - api/agents: attachment-resolution table + trajectory export endpoint (#3025 / #3094 / #3097). - api/system: POST /api/tools/{name}/invoke endpoint + fail-closed allowlist explanation. - cli/commands: 'Slash Commands in Chat' section listing CLI registry commands and pointing at channel commands. - providers/tools: 'CLI logins as first-class default providers' detection priority and override. - troubleshooting: 'Audit checks' subsection + table. - sdk: 'Auto-generated from openapi.json' section. * docs: refresh /new slash-command semantics (#3071 review fix) PR review on #3189 (houko) caught two stale references describing the pre-#3071 behavior of /new ('wipe the session' / 'clear history'). The new semantics fork off a fresh session while the previous one stays resumable on the same channel — phrasing updated in 4 places (en + zh × 2 files), plus a one-line clarifier in the slash-command roster section so the cross-reference doesn't silently inherit the old mental model: - configuration/channels: 'wipe the session' → 'fork off into a new session (the prior conversation stays resumable on the channel)' - integrations/api/realtime: '(clear history)' → 'forks into a fresh session — the prior session is preserved and remains resumable' - integrations/cli/commands roster paragraph: explicit note that /new is a fork, not a wipe. Same wording mirrored to docs/src/app/zh/. * docs(channels): 9 channel-related Apr 22-25 PRs (en + zh) Covers the channel + bridge feature wave: - #3020 per-agent ChannelOverrides — fixes the contradicting 'without modifying the agent manifest' phrasing in both en and zh. - #3077 prefix_agent_name + Signal plain-text default - #3009 Slack reactions_enabled + #3005 Feishu @mention preservation consolidated into a 'Reactions and Processing State' section - #3008 Signal media attachments - #3012 WhatsApp send_voice + dm_policy / group_policy - #3011 Webhook deliver_only mode - #2972 channel file downloads (file_download_dir / max_bytes config) * docs(config-core): provider timeout / browser CDP / cron session caps (en + zh) Three new `config.toml` sections that landed Apr 22-23: - #3004 `[provider_request_timeout_secs]` map - #2993 / #2991 `[browser].cdp_endpoint` + `cdp_auth_token_env` - #2994 `[kernel].cron_session_max_tokens` / `cron_session_max_messages` * docs(api): GET /attach + Owner Notice envelope (en + zh) - #3078 SSE attach endpoint for multi-client co-watching — describes the per-session SessionStreamHub fan-out, attacher behaviour, and the curl example. - #2965 ReplyEnvelope.owner_notice + notify_owner tool — message body field, StreamEvent variant, WhatsApp OWNER_JIDS fan-out. * docs(observability + ops): Tempo stack, cache_hit_ratio metric, daemon log tee (en + zh) - #3064 + #3149 — bundled observability stack (otel-collector / Tempo / Grafana), bring-up commands, [telemetry] config, business spans, cache_hit_ratio metric formula and where it surfaces - #3022 — `librefang start --foreground` now tees logs to ~/.librefang/logs/daemon-YYYY-MM-DD.log; same-day append; 7-day rotation pruned on daemon start * docs: hooks transform / lazy tools / mcp taint / capability detection / providers extras Final batch of catch-up docs covering the remaining Apr 22-25 user-facing features: - agent/hooks: transform_tool_result hook contract (#3003) - agent/templates: tool_search / tool_load lazy loading + lazy_tools opt-out (#3047) - integrations/mcp-a2a: TaintRuleId enum + per-tool skip_rules policy (#2999) - configuration/features: compaction summaries in user's conversation language (#3007) - configuration/providers/local: embedding auto-detection priority list (#3099) + Ollama capability detection (Modality / metadata pipeline, #3074 / #3133 / #3134 / #3140) - configuration/providers/management: custom image-gen provider via generic OpenAI-compat driver (#2998) + Moonshot file-upload helper via /v1/files (#2966) All sections mirrored to docs/src/app/zh/. * docs: address houko fix-PR audit (8 items, en + zh) PR review caught 1 stale-doc contradiction + 7 undocumented behaviour changes from the fix-PR cohort. - #3114 — configuration/core: 'api_key empty = unauthenticated' was stale; now documents the loopback-only policy (non-loopback bypass was closed) - #3170 — features/observability: auto_start is opt-in; home_dir- scoped Docker labels; RAII cleanup - #2989 — api/agents POST /message: per-request session_id override - #3048 — Interrupting a Running Agent Turn: /stop cascades into agent_send subagents (recursive) - #3056 + #3000 — desktop: dedicated Connection screen via custom URI scheme with retry / open-in-browser / uninstall - #2952 — triggers: response routes to agent's home channel - #2955 — triggers: task_posted assignee_match = 'self' | 'any' - #3069 — providers/local: 60-second reprobe + manual 'Test connection' refresh All sections mirrored in docs/src/app/zh/.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Builds on PR-1 (#3133) by lighting up the persisted-cache and Ollama runtime-probe layers in the lookup pipeline. Still passive — no kernel call site invokes
resolve_model_metadatayet, so runtime behaviour is unchanged. PR-3 will wire the kernel and retire the uniform 200K default inagent_loop.rs:1285.ModelCatalog/api/showPR-2.5 will expand L4 to Anthropic
/v1/modelsand generic OpenAI-compat/v1/models/{id}.Signature change
resolve_model_metadatais nowasync(was sync in PR-1). Required for thespawn_blocking-wrapped fs IO in L3 and the reqwest-driven HTTP probe in L4. Also grows ahome_dir: &Pathargument so the cache file path is dependency-injected (kernel will passconfig.home_dirin PR-3). Since PR-1 had no callers in production code, the breaking change is contained to the test suite.Layer 3: persisted cache
<home_dir>/cache/model_metadata.json.{ version: u32, entries: HashMap<String, CacheEntry> }. Entries carrycontext_window,max_output_tokens,fetched_at(RFC3339),source(string tag for telemetry).provider|base_url|modeltriple — same model id under different providers / endpoints stays separated (e.g.claude-opus-4-7onanthropicvscopilot).fetched_at + ttlfall through to L4 instead of being returned..tmp, rename over the live file). Partial writes never corrupt an existing cache.tracing::warn— the cache file is never load-bearing for daemon startup.tokio::task::spawn_blockingso the runtime never stalls on slow disks (encrypted volumes, network mounts, …).Layer 4: Ollama
/api/showprobe_PROBE_TIMEOUT).parse_ollama_showprefers theparametersblock'snum_ctxline (the effective server cap) over the nominalmodel_info.*.context_length(the model's theoretical max). The model might support 128K but the server is configured for 16K — we want what the user can actually use.0would break downstreamContextBudgetmath; falling through to L5 is safer.looks_like_ollamamatches the literalollama/ollama-cloudprovider tags and anybase_urlcontaining:11434(canonical Ollama port).OnceLock<reqwest::Client>built fromlibrefang_http::proxied_clientso corporate-proxy environments work transparently. Initialised lazily on first probe.On a successful probe, the result is written back to L3 (best-effort — write failures log a warn but don't propagate) so subsequent boots skip the network roundtrip.
Other changes
MetadataRequestgrowsapi_key: Option<&'a str>(will be consumed by the Anthropic probe in PR-2.5).#[tokio::test]+ atempfile::TempDir-backedhome_dir. The test helperfresh_home()returns an ownedTempDirso cache files are cleaned up on Drop.MetadataSource::PersistedCacheandRuntimeProbeare no longer placeholders.Tests (9 new, 23 total)
layer_3_cache_round_triplayer_3_stale_cache_falls_throughlayer_3_corrupted_file_does_not_panicparse_ollama_show_prefers_num_ctxnum_ctx 32768) wins over nominal (262144)parse_ollama_show_falls_back_to_model_infonum_ctxis absent, first*.context_lengthis usedparse_ollama_show_returns_none_on_missing_fieldsnum_ctx, no*.context_length) → Noneparse_ollama_show_rejects_zeronum_ctx 0and*.context_length: 0both rejectedollama_detection_provider_or_portlooks_like_ollamamatches both literal tag and 11434 portcache_key_separates_endpoint_namespacesTest plan
cargo check --workspace --lib— cleancargo test -p librefang-runtime --lib model_metadata— 23 ok (14 PR-1 ported to async + 9 new)cargo clippy -p librefang-runtime --all-targets -- -D warnings— cleanresolve_model_metadatais wired into the kernel.Stack
Base: PR-1 (
feat/model-metadata-lookup-m1, #3133). When #3133 lands the diff againstmaincollapses to just this PR's increments.Follow-ups:
/v1/modelsand generic OpenAI-compat/v1/models/{id}branches.resolve_model_metadataintokernel/mod.rs:4058-4062and the other catalog-based context lookup sites; retireDEFAULT_CONTEXT_WINDOW = 200_000; teachmodel_catalog.rs:732's Ollama merge to stop writing the false131_072default.See
.plans/model-metadata-lookup.mdfor full design.