Skip to content

fix(models): delegate generic provider catalogs to Hermes CLI#1726

Closed
Michaelyklam wants to merge 1 commit intonesquena:masterfrom
Michaelyklam:feat/model-catalog-cli-source
Closed

fix(models): delegate generic provider catalogs to Hermes CLI#1726
Michaelyklam wants to merge 1 commit intonesquena:masterfrom
Michaelyklam:feat/model-catalog-cli-source

Conversation

@Michaelyklam
Copy link
Copy Markdown
Contributor

@Michaelyklam Michaelyklam commented May 5, 2026

Thinking Path

  • Hermes CLI owns the live provider registry and model catalog behavior.
  • WebUI should not drift from the CLI by preferring stale static snapshots for generic provider model groups.
  • Explicit user-configured provider model allowlists still need to win, but otherwise the picker should ask Hermes CLI first and keep static WebUI lists as fallback.

What Changed

  • Added live provider model catalog loading through hermes_cli.models.provider_model_ids().
  • Preserved WebUI provider alias handling before falling back to static _PROVIDER_MODELS entries.
  • Kept explicit configured provider models allowlists as the highest-priority source.
  • Added regression tests for CLI-first generic provider catalogs, static fallback behavior, and provider-prefix handling.

Refs #1240.

Why It Matters

The model picker stays aligned with the installed Hermes Agent/CLI catalog as providers evolve, while still remaining safe when CLI catalog lookup is unavailable.

Verification

  • GitHub Actions: test (3.11), test (3.12), and test (3.13) are passing on commit 91890abd.
  • Added targeted regression coverage in tests/test_issue1240_generic_cli_catalog_sync.py, tests/test_issue644.py, and tests/test_model_resolver.py.

Risks / Follow-ups

Model Used

OpenAI Codex / GPT-5.5 via Hermes Agent, with terminal/file tools and maintainer-autopilot PR stewardship.

@nesquena-hermes
Copy link
Copy Markdown
Collaborator

Summary

Pulled the branch and walked api/config.py plus the three test files. The shape is right: ask Hermes CLI's provider_model_ids() first, then fall back to WebUI's static _PROVIDER_MODELS, with explicit user-configured models allowlists still winning. This is one source-of-truth slice for #1240 — the agent owns the live registry, WebUI follows.

Code reference

The new priority order on api/config.py:2980-2998:

provider_cfg = cfg.get("providers", {}).get(pid, {})
raw_models = []

# User-configured model allowlists are explicit local
# source-of-truth and should still beat auto-discovery.
# Otherwise, ask Hermes CLI first so WebUI tracks the same
# live catalog as the agent/CLI picker; WebUI's static
# _PROVIDER_MODELS table is now a fallback only (#1240).
if isinstance(provider_cfg, dict) and "models" in provider_cfg:
    cfg_models = provider_cfg["models"]
    if isinstance(cfg_models, dict):
        raw_models = [{"id": k, "label": k} for k in cfg_models.keys()]
    elif isinstance(cfg_models, list):
        raw_models = [{"id": k, "label": k} for k in cfg_models]

if not raw_models:
    raw_models = _models_from_live_provider_ids(
        pid,
        _read_live_provider_model_ids(pid),
    )

if not raw_models:
    raw_models = copy.deepcopy(_PROVIDER_MODELS.get(pid, []))

That four-tier resolution (config allowlist → CLI live → static fallback → auto-detected) reads cleanly against what each layer is authoritative for.

Cross-repo agent contract

The CLI side at ~/.hermes/hermes-agent/hermes_cli/models.py:1905 is exactly the right entry point — provider_model_ids() already special-cases the live-fetch providers internally:

def provider_model_ids(provider: Optional[str], *, force_refresh: bool = False) -> list[str]:
    normalized = normalize_provider(provider)
    if normalized == "openrouter":
        return model_ids(force_refresh=force_refresh)
    if normalized == "openai-codex":
        # ...uses live ChatGPT catalog endpoint with OAuth token...
        return get_codex_model_ids(access_token=access_token)
    if normalized in {"copilot", "copilot-acp"}:
        try:
            live = _fetch_github_models(_resolve_copilot_catalog_api_key())
            if live:
                return live
        except Exception:
            pass
    if normalized == "nous":
        # ...live Nous Portal /models endpoint...

Calling that means WebUI now sees catalog updates for openrouter, codex, copilot, nous, stepfun, etc. without a WebUI release. Good cross-repo factoring.

Diagnosis

Three small observations:

  1. Alias handling at api/config.py:1985-1990. _resolve_provider_alias(pid) is tried as a secondary lookup, and the seen dedup happens across both candidates. That handles cases like WebUI's google / x-ai display IDs that Hermes CLI normalizes to google-gemini-cli / xai. Confirmed _resolve_provider_alias is defined at api/config.py:661.

  2. Failure shape. _read_live_provider_model_ids() returns [] on any exception — no logging at info/warn level, only logger.debug("Failed to load %s models from hermes_cli", candidate). That is the correct level for a normal-path fallback (e.g. older agent installs without provider_model_ids); a noisier logger would spam logs for users on stale agents. Static fallback then takes over.

  3. Ollama label formatter routing. _models_from_live_provider_ids at api/config.py:2007-2020 selects _format_ollama_label for ollama and ollama-cloud. That preserves the existing label shape (e.g. Llama 3.1 70B) for live-discovered Ollama tags rather than falling back to the generic _get_label_for_model. Worth keeping in mind that any future Ollama label refinement needs to land in both the live and static paths now.

One question

The provider_cfg allowlist branch builds raw_models from raw IDs but does not apply any label transform. Is that intentional? For ollama, a configured allowlist will get [{"id": "llama3:70b", "label": "llama3:70b"}], while the CLI-live path will go through _format_ollama_label and produce [{"id": "llama3:70b", "label": "Llama 3 70B"}]. Probably already true on master, so not a regression — but if you wanted parity, plumbing the formatter into the allowlist branch is a one-liner. Not blocking.

Verification

The three test files cover the right cases: test_issue1240_generic_cli_catalog_sync.py for the new CLI-first ordering, test_issue644.py for prefix handling, and test_model_resolver.py adjustments for the new resolution order. CI is green on 3.11/3.12/3.13 at commit 91890ab. Looks ready to merge.

@nesquena-hermes
Copy link
Copy Markdown
Collaborator

Thanks @Michaelyklam — this shipped in v0.51.8 (commit 85d0279) as part of a 7-PR full-sweep batch release. Stage rebased your branch onto current master, ran the full pre-release gate (4584 pytest, browser tests, Opus advisor verdict SHIP), and merged via release PR #1737.

GitHub didn't auto-close because the merge commit only references the squash-merged stage branch, not your fork's commit directly — closing manually for hygiene.

Live now on https://get-hermes.ai/ and on existing installs after git pull + restart.

Release notes: https://github.com/nesquena/hermes-webui/releases/tag/v0.51.8

pull Bot pushed a commit to soitun/hermes-webui that referenced this pull request May 6, 2026
pull Bot pushed a commit to soitun/hermes-webui that referenced this pull request May 6, 2026
…p + test-isolation fix

Constituent PRs:
- nesquena#1725 (@Michaelyklam) — simplify compact Activity row summary
- nesquena#1726 (@Michaelyklam) — delegate generic provider catalogs to Hermes CLI (slice of nesquena#1240)
- nesquena#1727 (@Michaelyklam) — link Claude Code OAuth in onboarding (closes nesquena#1362)
- nesquena#1728 (@starship-s) — preserve profile context when starting chats
- nesquena#1729 (@Michaelyklam) — persist compact Activity disclosure state
- nesquena#1730 (@Michaelyklam) — prevent sticky sidebar hover drag state
- nesquena#1732 (@Sanjays2402) — unpin scroll on small upward motion during streaming (closes nesquena#1731)

Plus 2 in-stage absorbed fixes:
- test-isolation fix: monkeypatch.setattr(config, 'cfg', X) survives PR nesquena#1728's
  path/mtime-aware get_config() reload. Mandatory before tag (Opus stage-302).
- Opus SHOULD-FIX #1: _lastScrollTop reset on session switch (nesquena#1732 follow-up).

Tests: 4537 → 4584 passing (+47). 0 regressions. Full suite ~128s. Stably green.

Pre-release verification:
- All 7 PRs CI-green individually + rebased onto master
- pytest 4584 passed, 0 failed (multiple runs)
- node -c clean on all 4 modified .js files
- 11/11 browser API endpoints PASS on isolated port 8789
- 20 QA tests via webui_qa_agent.sh PASS
- Opus advisor: SHIP, 5/5 verification clean, 0 MUST-FIX, 1 SHOULD-FIX absorbed
  (_lastScrollTop reset), 1 SHOULD-FIX deferred (nesquena#1736 — _clear_anthropic_env_values
  race, onboarding-time-only)

Closes nesquena#1362, nesquena#1731.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants