Skip to content

feat(usai-catalog): carry verified capability metadata through a catalog refresh #576

Closed

Description

@mogul

🤖: AI agent (First Mate), filing on behalf of @mogul.

Goal

A refresh of the USAi catalog must not erase verified reasoning / inputModalities values.

Problem

#541 (data PR #575) commits verified inputModalities for 12 models, with an evidence file. The refresh entrypoint in #573 rebuilds catalog.json from the live feeds, and the builder drops capability values unless the unverified --include-capabilities copy is requested. A plain refresh after #575 would therefore silently remove the verified values (and a refresh using --include-capabilities would replace them with unverified upstream values). The PR body and README of #575 flag this; nothing enforces it yet.

Scope

  • Treat verified capability values as curated overlay data keyed by model id: carry them forward on refresh, as recommendations are already carried forward.
  • Drop a model's values only when the model leaves the feed, and report that as drift.
  • Never enable the unverified models.dev copy path by default; if --include-capabilities is used, do not let it overwrite a verified value without an explicit, visible conflict.
  • Keep sources.capabilitiesVerified (the release reminder) consistent when values are carried forward.
  • Report added, changed and dropped capability values as drift in --check mode.

Acceptance

Offline fixtures show that a refresh preserves verified values, drops values for removed models, reports changes as drift, and refuses to overwrite a verified value from the unverified path.

Trigger

After #575 (the data change for #541) and #573 (the refresh entrypoint) merge. Child slice of #419.

AI-assisted tracker entry (OpenCode). Human review required.

Activity

  1. mogul commented on Nov 5, 2026

    @mogul
    ContributorAuthor

    🤖: AI agent (First Mate), posting on behalf of @mogul. Closing as completed.

    Delivered in #578 (merged). Verified reasoning / inputModalities values are carried forward through a catalog refresh as overlay data keyed by model id, and are dropped only when a model leaves the feed (reported as drift). --include-capabilities is refused before any write while verified values are carried (conflict or mixed), sources.capabilitiesVerified is carried and never advanced, and --check reports capability added, changed and dropped lines for the catalog and the neutral snapshot.

    Cross-checked on the data branch of #575: the offline check reports up to date and a write-mode refresh keeps all 12 verified values byte-identical.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions