Skip to content

tracking: provider/model inventory has no scriptable surface — five PRs reinvent it, four issues blocked #23359

Description

@kshitijk4poor

The gap

Hermes can list its providers and models four ways today — cli.py interactive picker, tui_gateway/server.py model.options, hermes_cli/web_server.py /api/model/options, hermes_cli/main.py:2250 switcher — but none of them are scriptable. Every entry point is either interactive (TTY required) or HTTP (gateway running). There is no way for a cron job, a CI script, an external dashboard, or a Hermes-based agent to ask "what providers exist on this host, which are configured, what models can I switch to" without parsing free-form text or maintaining a reverse-engineered mental model of hermes_cli/auth.py:PROVIDER_REGISTRY.

The dashboard at /api/model/options already returns this exact data as JSON — it just isn't reachable from the CLI.

Concrete pain

  • Cron job verifying a key is valid before a batch runs has to spawn an interactive hermes, parse hermes doctor's free-form output, or import internals.
  • Observability layer can't answer "of 33 supported providers, which 12 are working on this host" without standing up the gateway HTTP server.
  • Multi-host fleet check is ssh + hermes model + eyeball + repeat instead of ssh + hermes providers list --json | jq | aggregate.
  • Hermes-based AI agent introspecting its own environment can't scrape an answer — has to query OpenAI/Anthropic/etc. directly and bypass the framework it's running on.
  • Plugin authors validating a new plugins/model-providers/<name>/ have to launch the picker and scroll, instead of hermes providers list | grep <name>.
  • Debugging "why isn't my provider showing up" needs reading code; with a scriptable surface, hermes providers list would show a row with auth_state: unconfigured, env_vars: TOGETHER_API_KEY (not set) in five seconds.

Five open PRs reinvent the same traversal

Each of the following touches inventory traversal for one narrow purpose, with no shared function:

Four issues converge on the same gap:

Empirical findings worth tracking separately

While researching the fix I verified these against main at d62808c37. They are not blockers for the inventory CLI, but should be tracked separately so they don't get lost:

  1. Four registries don't fully overlap. providers.list_providers() = 33 plugin profiles. hermes_cli.auth.PROVIDER_REGISTRY = 33 (different set — excludes openrouter, custom; includes lmstudio, tencent-tokenhub). hermes_cli.models.CANONICAL_PROVIDERS = 35 (superset). _PROVIDER_MODELS = 32 (yet another set: includes moonshot, openai; missing custom, ollama-cloud, openrouter, qwen-oauth). Iterating over any single one drops real providers from the operator-visible inventory.

  2. profile.fetch_models() returns models the runtime rejects. OpenRouter raw = 367. provider_model_ids("openrouter") (used by the picker) = 34 tool-call-filtered. The 333-model gap = models that don't support tool-calling. The plugin's own docstring says: "Tool-call capability filtering is applied by hermes_cli/models.py via fetch_openrouter_models() → _openrouter_model_supports_tools(), not here." Anyone (including agents introspecting via get_provider_profile) calling profile.fetch_models() directly gets the unfiltered list and runtime errors on most picks.

  3. profile.fallback_models is smaller than _PROVIDER_MODELS[slug] for 28 of 33 providers. anthropic curated = 8, profile.fallback_models = 0. copilot = 17/0. bedrock = 10/0. nous = 24/2 — and the 2 in profile aren't in curated.

  4. provider_model_ids("nous") without credentials bypasses the remote catalog. Returns 24 (static _PROVIDER_MODELS). get_curated_nous_model_ids() returns 32 (with the remote manifest). Same gap may exist for other manifest-backed providers.

  5. list_authenticated_providers does unconditional live fetches before per-provider gating: fetch_models_dev() (HTTP to models.dev/api.json, cached 1h), fetch_ollama_cloud_models() (always), fetch_lmstudio_models() (when LM_BASE_URL set or current = lmstudio). Callers assuming "this is offline-safe" hit the network.

Proposed scope of this tracking issue

A non-interactive, scriptable, TTY-free CLI surface that consolidates the four current surfaces:

hermes models list [--provider NAME] [--json] [--all] [--no-live] [--offline]
hermes models status [--provider NAME] [--json] [--probe] [--offline]
hermes providers list [--json] [--all] [--offline]

Default behaviour matches the existing picker/dashboard (configured-only, network calls allowed, cached). --offline synthesises from in-repo static data only — no HTTP, no remote catalogs, no credential-pool probing — for cron/CI use. --probe is opt-in and explicitly mutually exclusive with --offline.

Fixing this lands as:

Out of scope (tracked separately)

References

Metadata

Metadata

Assignees

No one assigned

    Labels

    P2Medium — degraded but workaround existscomp/cliCLI entry point, hermes_cli/, setup wizardtype/featureNew feature or request

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions