Skip to content

feat(agents): llm provider cutover#2567

Merged
jordan-umusu merged 29 commits intomainfrom
feat/llm-providers-v2-integration
Apr 30, 2026
Merged

feat(agents): llm provider cutover#2567
jordan-umusu merged 29 commits intomainfrom
feat/llm-providers-v2-integration

Conversation

@jordan-umusu
Copy link
Copy Markdown
Collaborator

@jordan-umusu jordan-umusu commented Apr 24, 2026

Summary by cubic

Cut over to LLM Providers v2 with an org-wide catalog, per-workspace enablement, and a unified model picker. Models now resolve by catalog id across agents, presets, workflows, MCP tools, ranker, and SDK; credentials resolve per catalog row; passthrough URLs preserve version suffixes; legacy workflow history/tokens remain compatible.

  • New Features

    • Platform catalog is validated and seeded on startup via a resilient loader; new API routers added for catalog, access, and custom providers; admin “Agent” page with search and provider icons (OpenAI, Microsoft/Azure, Anthropic, Bedrock, Ollama, vLLM).
    • Workspace Settings → “AI models” lets you enable/disable catalog models and set a default; org defaults apply with workspace overrides; default model is set and returned by catalog id and gated by agent_addons.
    • Catalog and custom provider CRUD store encrypted key‑values; credentials dialog supports create/update; legacy custom‑provider updates fall back to migrated encrypted base_url when present.
    • Gateway caches credentials by catalog id with org/workspace fallback for legacy tokens; passthrough proxy normalizes upstream URLs and preserves upstream “/vN” suffix; credential scope flags removed in favor of catalog id.
    • Unified model selection: a single ModelSelection (AgentModel UI) is used across builder actions, presets, chat, MCP tools, and the ranker; SDK/DSL/runtime carry catalog_id end‑to‑end; preset creation rejects disabled catalog ids.
  • Migration

    • Remove any usage of use_workspace_credentials; legacy workflow history and LLM tokens are tolerated via relaxed schema parsing.
    • Update workflows/registry actions to one model selection (model: ModelSelection) instead of model_name/model_provider; enable desired models in Workspace → “AI models”, then update presets to pick catalog‑backed models.
    • Set the organization’s default model by catalog id; workspace model settings remain gated by the agent_addons entitlement.

Written for commit 8726094b5d1ffb94fe98eefb5443133ee8a5213a. Summary will update on new commits. Review in cubic

@zeropath-ai
Copy link
Copy Markdown

zeropath-ai Bot commented Apr 24, 2026

No security or compliance issues detected. Reviewed everything up to 4684104.

Security Overview
Detected Code Changes

The diff is too large to display a summary of code changes.

@jordan-umusu jordan-umusu force-pushed the feat/llm-providers-v2-migrations branch from 3bd0de9 to 97ed3cd Compare April 24, 2026 22:50
@jordan-umusu jordan-umusu force-pushed the feat/llm-providers-v2-integration branch from 71eeb5e to 3eabb75 Compare April 24, 2026 22:52
@blacksmith-sh

This comment has been minimized.

@jordan-umusu jordan-umusu marked this pull request as ready for review April 25, 2026 00:43
Copy link
Copy Markdown
Contributor

@cubic-dev-ai cubic-dev-ai Bot left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

7 issues found across 72 files

Confidence score: 2/5

  • Merge risk is high because tracecat/agent/preset/activities.py appears to bypass workspace model-access checks, which can let disabled catalog models still resolve custom-provider runtime config (permission/control regression).
  • There is concrete user-facing correctness risk in model selection: frontend/src/components/builder/panel/action-panel-fields.tsx ignores catalog_id, so duplicate provider/model names may resolve to the wrong catalog option.
  • Several medium issues increase instability/perf risk (tracecat/agent/catalog/loader.py malformed JSON handling, frontend/src/components/organization/org-agent-credentials-dialog.tsx hidden required fields, frontend/src/hooks/use-admin.ts expensive refetch-on-keystroke, and frontend/src/components/settings/workspace-model-settings.tsx unsaved form resets).
  • Pay close attention to tracecat/agent/preset/activities.py, frontend/src/components/builder/panel/action-panel-fields.tsx, and tracecat/agent/catalog/loader.py - access-control bypass, ambiguous model resolution, and runtime failure paths are the most impactful concerns.
Prompt for AI agents (unresolved issues)

Check if these issues are valid — if so, understand the root cause of each and fix them. If appropriate, use sub-agents to investigate and fix each issue separately.


<file name="tracecat/agent/catalog/loader.py">

<violation number="1" location="tracecat/agent/catalog/loader.py:31">
P2: Validate `catalog_data`/`models` shapes before using `.get()` so malformed JSON structure falls back to an empty/filtered list instead of raising at runtime.</violation>
</file>

<file name="frontend/src/components/organization/org-agent-credentials-dialog.tsx">

<violation number="1" location="frontend/src/components/organization/org-agent-credentials-dialog.tsx:181">
P2: Hardcoded layout sections can hide provider-config fields, making new required credentials impossible to enter.</violation>
</file>

<file name="frontend/src/components/settings/workspace-model-settings.tsx">

<violation number="1" location="frontend/src/components/settings/workspace-model-settings.tsx:255">
P2: Avoid unconditional `form.reset` in the data-sync effect; it can wipe unsaved model selections when queries refetch.</violation>
</file>

<file name="frontend/src/components/builder/panel/action-panel-fields.tsx">

<violation number="1" location="frontend/src/components/builder/panel/action-panel-fields.tsx:1039">
P2: Model selection resolution is ambiguous because it ignores `catalog_id`, so duplicate provider/model-name entries can map to the wrong catalog option.</violation>
</file>

<file name="frontend/src/hooks/use-admin.ts">

<violation number="1" location="frontend/src/hooks/use-admin.ts:684">
P2: Including `query` in the React Query key causes a full re-fetch of all catalog pages on every search keystroke, which is unnecessarily expensive.</violation>
</file>

<file name="tracecat/agent/preset/activities.py">

<violation number="1" location="tracecat/agent/preset/activities.py:173">
P1: Catalog credential resolution bypasses workspace model-access checks, allowing disabled catalog models to still resolve custom-provider runtime config.</violation>
</file>

<file name="tests/unit/test_agent_management_service.py">

<violation number="1" location="tests/unit/test_agent_management_service.py:617">
P3: Add `@pytest.mark.anyio` so this async test is actually executed.</violation>
</file>

Reply with feedback, questions, or to request a fix. Tag @cubic-dev-ai to re-run a review.

Comment thread tracecat/agent/preset/activities.py Outdated
Comment thread tracecat/agent/catalog/loader.py
Comment thread frontend/src/components/organization/org-agent-credentials-dialog.tsx Outdated
Comment thread frontend/src/components/settings/workspace-model-settings.tsx Outdated
Comment thread frontend/src/components/builder/panel/action-panel-fields.tsx
Comment thread frontend/src/hooks/use-admin.ts Outdated
Comment thread tests/unit/test_agent_management_service.py
Copy link
Copy Markdown

@chatgpt-codex-connector chatgpt-codex-connector Bot left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: 8aa2e70be2

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

Comment thread tracecat/agent/preset/activities.py Outdated
@jordan-umusu jordan-umusu force-pushed the feat/llm-providers-v2-migrations branch from 97ed3cd to 3dd2d98 Compare April 25, 2026 13:04
@jordan-umusu jordan-umusu force-pushed the feat/llm-providers-v2-integration branch from 8aa2e70 to 8233c8c Compare April 25, 2026 13:04
Copy link
Copy Markdown

@chatgpt-codex-connector chatgpt-codex-connector Bot left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: 8233c8c04b

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

Comment thread tracecat/agent/preset/service.py Outdated
Comment thread tracecat/agent/service.py
@jordan-umusu jordan-umusu force-pushed the feat/llm-providers-v2-migrations branch 9 times, most recently from f7c2084 to b1cead5 Compare April 27, 2026 15:50
@jordan-umusu jordan-umusu force-pushed the feat/llm-providers-v2-integration branch from 8233c8c to 5424eb0 Compare April 27, 2026 16:22
Copy link
Copy Markdown

@chatgpt-codex-connector chatgpt-codex-connector Bot left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: 5424eb07b2

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

Comment thread tracecat/agent/service.py Outdated
Comment thread packages/tracecat-registry/tracecat_registry/core/ai.py
Comment thread tracecat/mcp/server.py Outdated
@jordan-umusu jordan-umusu force-pushed the feat/llm-providers-v2-integration branch from 5424eb0 to e955f9c Compare April 27, 2026 19:31
Copy link
Copy Markdown

@chatgpt-codex-connector chatgpt-codex-connector Bot left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: e955f9c0de

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

Comment thread tracecat/agent/preset/service.py Outdated
Comment thread tracecat/agent/service.py
@daryllimyt
Copy link
Copy Markdown
Contributor

Non-blocking — small typing tightening on _CLOUD_PROVIDER_TARGET_KEYS and the two helpers that read it in tracecat/agent/service.py.

Why:

  • tuple[str, str] for the inner pair is opaque; readers have to remember which is which.
  • dict[str, ...] permits _CLOUD_PROVIDER_TARGET_KEYS.get("bogus") statically (always None at runtime).
  • A Literal/TypeGuard pair gives a typed [] lookup and a name we can reuse downstream.

Take it or leave it:

 from collections.abc import AsyncIterator, Iterator
 from dataclasses import replace
 from datetime import UTC, datetime
+from typing import Literal, NamedTuple, TypeGuard

-_CLOUD_PROVIDER_TARGET_KEYS: dict[str, tuple[tuple[str, str], ...]] = {
+CloudProviderSlug = Literal["bedrock", "azure_openai", "azure_ai", "vertex_ai"]
+
+
+class CloudTargetKey(NamedTuple):
+    metadata_key: str
+    credential_key: str
+
+
+_CLOUD_PROVIDER_TARGET_KEYS: dict[CloudProviderSlug, tuple[CloudTargetKey, ...]] = {
     "bedrock": (
-        ("inference_profile_id", "AWS_INFERENCE_PROFILE_ID"),
-        ("model_id", "AWS_MODEL_ID"),
+        CloudTargetKey("inference_profile_id", "AWS_INFERENCE_PROFILE_ID"),
+        CloudTargetKey("model_id", "AWS_MODEL_ID"),
     ),
-    "azure_openai": (("deployment_name", "AZURE_DEPLOYMENT_NAME"),),
-    "azure_ai": (("azure_ai_model_name", "AZURE_AI_MODEL_NAME"),),
-    "vertex_ai": (("vertex_model", "VERTEX_AI_MODEL"),),
+    "azure_openai": (CloudTargetKey("deployment_name", "AZURE_DEPLOYMENT_NAME"),),
+    "azure_ai": (CloudTargetKey("azure_ai_model_name", "AZURE_AI_MODEL_NAME"),),
+    "vertex_ai": (CloudTargetKey("vertex_model", "VERTEX_AI_MODEL"),),
 }


+def _is_cloud_provider(slug: str) -> TypeGuard[CloudProviderSlug]:
+    return slug in _CLOUD_PROVIDER_TARGET_KEYS
+
+
     def _catalog_target_credentials(self, row: AgentCatalog) -> dict[str, str]:
         """Project cloud catalog target metadata into runtime credential keys."""
-        target_keys = _CLOUD_PROVIDER_TARGET_KEYS.get(row.model_provider)
-        if not target_keys:
+        if not _is_cloud_provider(row.model_provider):
             return {}
+        target_keys = _CLOUD_PROVIDER_TARGET_KEYS[row.model_provider]
         metadata = row.model_metadata if isinstance(row.model_metadata, dict) else {}
         credentials: dict[str, str] = {}
-        for metadata_key, credential_key in target_keys:
-            value = metadata.get(metadata_key)
+        for spec in target_keys:
+            value = metadata.get(spec.metadata_key)
             if isinstance(value, str) and value:
-                credentials[credential_key] = value
+                credentials[spec.credential_key] = value
         return credentials

     def _catalog_encrypted_target_credentials(
         self,
         row: AgentCatalog,
     ) -> dict[str, str]:
         """Decrypt migrated catalog config for cloud target keys only."""
-        target_keys = _CLOUD_PROVIDER_TARGET_KEYS.get(row.model_provider)
-        if not target_keys or row.encrypted_config is None:
+        if not _is_cloud_provider(row.model_provider) or row.encrypted_config is None:
             return {}
-        allowed_keys = {credential_key for _, credential_key in target_keys}
+        target_keys = _CLOUD_PROVIDER_TARGET_KEYS[row.model_provider]
+        allowed_keys = {spec.credential_key for spec in target_keys}

Verified clean under basedpyright tracecat/agent/service.py (0 errors, 0 warnings).

@jordan-umusu jordan-umusu requested a review from daryllimyt April 27, 2026 20:47
@jordan-umusu jordan-umusu added the agents LLM agents label Apr 27, 2026
Copy link
Copy Markdown
Contributor

@cubic-dev-ai cubic-dev-ai Bot left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

1 issue found across 9 files (changes from recent commits).

Prompt for AI agents (unresolved issues)

Check if these issues are valid — if so, understand the root cause of each and fix them. If appropriate, use sub-agents to investigate and fix each issue separately.


<file name="tracecat/mcp/server.py">

<violation number="1" location="tracecat/mcp/server.py:605">
P1: Custom provider validation uses org-secret retrieval that ignores v2 `AgentCustomProvider` rows, so valid custom providers can be rejected as unconfigured.</violation>
</file>

Reply with feedback, questions, or to request a fix. Tag @cubic-dev-ai to re-run a review.

Comment thread tracecat/mcp/server.py Outdated
Copy link
Copy Markdown

@chatgpt-codex-connector chatgpt-codex-connector Bot left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: c5146432e3

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

Comment thread tracecat/agent/preset/activities.py
Comment thread tracecat/agent/internal_router.py
Comment thread tracecat/mcp/server.py Outdated
@jordan-umusu jordan-umusu force-pushed the feat/llm-providers-v2-integration branch from c514643 to 75e1f3f Compare April 27, 2026 21:57
Copy link
Copy Markdown

@chatgpt-codex-connector chatgpt-codex-connector Bot left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: 75e1f3f63f

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

Comment thread tracecat/agent/gateway.py
@jordan-umusu jordan-umusu force-pushed the feat/llm-providers-v2-integration branch from 3a6faa7 to 4684104 Compare April 30, 2026 20:35
@jordan-umusu jordan-umusu temporarily deployed to internal-registry-ci April 30, 2026 20:35 — with GitHub Actions Inactive
@jordan-umusu jordan-umusu temporarily deployed to internal-registry-ci April 30, 2026 20:36 — with GitHub Actions Inactive
@jordan-umusu jordan-umusu merged commit a43de6b into main Apr 30, 2026
18 checks passed
@jordan-umusu jordan-umusu deleted the feat/llm-providers-v2-integration branch April 30, 2026 20:41
Copy link
Copy Markdown

@chatgpt-codex-connector chatgpt-codex-connector Bot left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

💡 Codex Review

model_name=version.model_name,
model_provider=version.model_provider,
base_url=version.base_url,
output_type=cast(OutputType | None, version.output_type),

P2 Badge Include catalog_id when serializing preset versions

AgentPresetVersionRead now carries catalog_id, but build_version_read() never sets it, so every version API response reports catalog_id as null even for catalog-backed versions. That drops model-binding metadata for clients consuming /agent-presets/.../versions/..., and any client that round-trips a fetched version into an update/create flow can unintentionally lose catalog-backed execution and fall back to legacy provider-secret behavior.

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

agents LLM agents

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants