roadmap: scale Tier 2 overlap detection for registry-sized catalogs #41

Context

OpenClaw feedback identified that the current Tier 2 local-catalog workflow cannot operate at ClawHub scale.

The current implementation is intentionally bounded:

• direct collection discovery is capped at 256 entries;
• local catalogs are capped at 5,000 entries;
• collection duplicate detection performs bounded pairwise comparisons in memory;
• catalogs are loaded from a local JSON artifact rather than maintained incrementally.

These limits are appropriate for untrusted local inputs, but they make the current design unsuitable for a large, continuously changing public registry.

Goal

Add a registry-scale overlap-detection architecture based on a persistent, incrementally maintained vector index and top-k nearest-neighbor retrieval. Preserve the existing bounded local-catalog workflow for local and CI use.

Requirements

• Define the supported registry size, update rate, query latency, cost, and recall targets.
• Support incremental insert, update, delete, and version replacement without rebuilding the full index.
• Query top-k candidates without loading or comparing the entire catalog.
• Preserve content fingerprints, provider/model/mode compatibility metadata, and deterministic skill identity.
• Keep explicit resource ceilings and fail-closed validation for untrusted inputs.
• Provide an index migration/versioning strategy and operational observability.
• Add integration and performance tests using a representative registry-scale corpus.
• Document how the scalable service relates to the existing local JSON catalog.

Acceptance criteria

• Agreed registry-scale capacity and latency targets are documented.
• A target architecture and storage/index backend are selected.
• Incremental upsert and deletion are supported.
• Queries return bounded top-k candidates without an all-pairs scan.
• Compatibility and provenance metadata are retained and validated.
• Scale, update, and regression tests cover the agreed operating envelope.
• Existing local-catalog behavior remains supported or has a documented migration path.

Out of scope

Automatic plagiarism or policy-enforcement verdicts. This issue provides the scalable retrieval foundation; copied-skill adjudication should require separate provenance and verification policy.