feat(provider)!: sync catalog from models.dev + fix discovery schema#41
Merged
Conversation
Add a catalog-sync toolchain that refreshes the embedded catalog.json from
the live models.dev API, and fix the discovery parser which never matched
the real models.dev schema.
- catalog-sync feature + llm-kernel-sync-catalog binary: fetch models.dev,
merge with field precedence, --check drift detection, atomic write
- provider/mapping: catalog->models.dev provider-id mapping (8 exact,
7 aliased, 5 manual)
- provider/sync: pure merge_catalog + CatalogDiff/PriceDelta; provider
service fields catalog-wins, model data models.dev-wins, empty
api_base_url/npm_package/doc_url filled from upstream
- ProviderIndex::from_providers + with_discovered (discovery-gated):
overlay runtime-discovered models so find_model/estimate_cost see them
- discovery: rewrite models.dev parser to the REAL provider-keyed map
schema; enrich ModelEntry with cost/modalities/capabilities + Default;
add fetch/fetch_from; From<ModelEntry> for ModelDescriptor
- catalog value types now derive Serialize + PartialEq
- catalog.json refreshed from models.dev: 20 providers, 351 models
- docs: README + 10 i18n READMEs updated for the new discovery API and
the runtime with_discovered path; AGENTS.md + CHANGELOG updated
BREAKING CHANGE: ModelsDevPayload is now a provider-keyed map mirroring
the real models.dev API (HashMap<provider_id, provider>). The previous
{ models: Vec<ModelEntry> } shape could not parse the live endpoint.
ModelEntry gained optional cost/modalities/capabilities/family/
release_date/knowledge fields (additive for serde reads).
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
catalog-syncfeature +llm-kernel-sync-catalogbinary that refreshes the embeddedcatalog.jsonfrom the live models.dev API (--checkdrift detection, atomic write, field-precedence merge).ProviderIndex::with_discovered()overlays runtime-discovered models soestimate_cost/find_modelsee them.{ models: [...] }shape never parsed the livehttps://models.dev/api.json.ModelEntryenriched with cost/modalities/capabilities.Field precedence (sync)
api_base_url/npm_package/doc_url: filled from models.dev.glm-5,ZAI_API_KEY, all existing tests preserved.Verification (local)
cargo test --all-features→ 502 passed, 0 failedcargo clippy --all-features -- -D warnings→ cleancargo fmt --all -- --check→ cleanRUSTDOCFLAGS=-D warnings cargo doc --all-features --no-deps→ cleancargo run --bin llm-kernel-sync-catalog --features catalog-sync -- --checkafter write → in sync (idempotent)--features catalog-syncand--no-default-featuresbuilds cleanBREAKING CHANGE
ModelsDevPayloadis now a provider-keyed map mirroring the real models.dev API (HashMap<provider_id, provider>). The previous{ models: Vec<ModelEntry> }shape could not parse the live endpoint.ModelEntrygained optional fields (additive for serde reads). → argues for a minor bump (0.10.0) rather than patch.Docs
README + 10 i18n READMEs updated (discovery API + new "Keeping the catalog fresh" runtime-merge section +
catalog-syncfeature row + counts).