feat(agents): llm provider cutover#2567
Conversation
|
✅ No security or compliance issues detected. Reviewed everything up to 4684104. Security OverviewDetected Code ChangesThe diff is too large to display a summary of code changes. |
3bd0de9 to
97ed3cd
Compare
71eeb5e to
3eabb75
Compare
This comment has been minimized.
This comment has been minimized.
There was a problem hiding this comment.
7 issues found across 72 files
Confidence score: 2/5
- Merge risk is high because
tracecat/agent/preset/activities.pyappears to bypass workspace model-access checks, which can let disabled catalog models still resolve custom-provider runtime config (permission/control regression). - There is concrete user-facing correctness risk in model selection:
frontend/src/components/builder/panel/action-panel-fields.tsxignorescatalog_id, so duplicate provider/model names may resolve to the wrong catalog option. - Several medium issues increase instability/perf risk (
tracecat/agent/catalog/loader.pymalformed JSON handling,frontend/src/components/organization/org-agent-credentials-dialog.tsxhidden required fields,frontend/src/hooks/use-admin.tsexpensive refetch-on-keystroke, andfrontend/src/components/settings/workspace-model-settings.tsxunsaved form resets). - Pay close attention to
tracecat/agent/preset/activities.py,frontend/src/components/builder/panel/action-panel-fields.tsx, andtracecat/agent/catalog/loader.py- access-control bypass, ambiguous model resolution, and runtime failure paths are the most impactful concerns.
Prompt for AI agents (unresolved issues)
Check if these issues are valid — if so, understand the root cause of each and fix them. If appropriate, use sub-agents to investigate and fix each issue separately.
<file name="tracecat/agent/catalog/loader.py">
<violation number="1" location="tracecat/agent/catalog/loader.py:31">
P2: Validate `catalog_data`/`models` shapes before using `.get()` so malformed JSON structure falls back to an empty/filtered list instead of raising at runtime.</violation>
</file>
<file name="frontend/src/components/organization/org-agent-credentials-dialog.tsx">
<violation number="1" location="frontend/src/components/organization/org-agent-credentials-dialog.tsx:181">
P2: Hardcoded layout sections can hide provider-config fields, making new required credentials impossible to enter.</violation>
</file>
<file name="frontend/src/components/settings/workspace-model-settings.tsx">
<violation number="1" location="frontend/src/components/settings/workspace-model-settings.tsx:255">
P2: Avoid unconditional `form.reset` in the data-sync effect; it can wipe unsaved model selections when queries refetch.</violation>
</file>
<file name="frontend/src/components/builder/panel/action-panel-fields.tsx">
<violation number="1" location="frontend/src/components/builder/panel/action-panel-fields.tsx:1039">
P2: Model selection resolution is ambiguous because it ignores `catalog_id`, so duplicate provider/model-name entries can map to the wrong catalog option.</violation>
</file>
<file name="frontend/src/hooks/use-admin.ts">
<violation number="1" location="frontend/src/hooks/use-admin.ts:684">
P2: Including `query` in the React Query key causes a full re-fetch of all catalog pages on every search keystroke, which is unnecessarily expensive.</violation>
</file>
<file name="tracecat/agent/preset/activities.py">
<violation number="1" location="tracecat/agent/preset/activities.py:173">
P1: Catalog credential resolution bypasses workspace model-access checks, allowing disabled catalog models to still resolve custom-provider runtime config.</violation>
</file>
<file name="tests/unit/test_agent_management_service.py">
<violation number="1" location="tests/unit/test_agent_management_service.py:617">
P3: Add `@pytest.mark.anyio` so this async test is actually executed.</violation>
</file>
Reply with feedback, questions, or to request a fix. Tag @cubic-dev-ai to re-run a review.
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: 8aa2e70be2
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
97ed3cd to
3dd2d98
Compare
8aa2e70 to
8233c8c
Compare
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: 8233c8c04b
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
f7c2084 to
b1cead5
Compare
8233c8c to
5424eb0
Compare
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: 5424eb07b2
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
5424eb0 to
e955f9c
Compare
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: e955f9c0de
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
|
Non-blocking — small typing tightening on Why:
Take it or leave it: from collections.abc import AsyncIterator, Iterator
from dataclasses import replace
from datetime import UTC, datetime
+from typing import Literal, NamedTuple, TypeGuard
-_CLOUD_PROVIDER_TARGET_KEYS: dict[str, tuple[tuple[str, str], ...]] = {
+CloudProviderSlug = Literal["bedrock", "azure_openai", "azure_ai", "vertex_ai"]
+
+
+class CloudTargetKey(NamedTuple):
+ metadata_key: str
+ credential_key: str
+
+
+_CLOUD_PROVIDER_TARGET_KEYS: dict[CloudProviderSlug, tuple[CloudTargetKey, ...]] = {
"bedrock": (
- ("inference_profile_id", "AWS_INFERENCE_PROFILE_ID"),
- ("model_id", "AWS_MODEL_ID"),
+ CloudTargetKey("inference_profile_id", "AWS_INFERENCE_PROFILE_ID"),
+ CloudTargetKey("model_id", "AWS_MODEL_ID"),
),
- "azure_openai": (("deployment_name", "AZURE_DEPLOYMENT_NAME"),),
- "azure_ai": (("azure_ai_model_name", "AZURE_AI_MODEL_NAME"),),
- "vertex_ai": (("vertex_model", "VERTEX_AI_MODEL"),),
+ "azure_openai": (CloudTargetKey("deployment_name", "AZURE_DEPLOYMENT_NAME"),),
+ "azure_ai": (CloudTargetKey("azure_ai_model_name", "AZURE_AI_MODEL_NAME"),),
+ "vertex_ai": (CloudTargetKey("vertex_model", "VERTEX_AI_MODEL"),),
}
+def _is_cloud_provider(slug: str) -> TypeGuard[CloudProviderSlug]:
+ return slug in _CLOUD_PROVIDER_TARGET_KEYS
+
+
def _catalog_target_credentials(self, row: AgentCatalog) -> dict[str, str]:
"""Project cloud catalog target metadata into runtime credential keys."""
- target_keys = _CLOUD_PROVIDER_TARGET_KEYS.get(row.model_provider)
- if not target_keys:
+ if not _is_cloud_provider(row.model_provider):
return {}
+ target_keys = _CLOUD_PROVIDER_TARGET_KEYS[row.model_provider]
metadata = row.model_metadata if isinstance(row.model_metadata, dict) else {}
credentials: dict[str, str] = {}
- for metadata_key, credential_key in target_keys:
- value = metadata.get(metadata_key)
+ for spec in target_keys:
+ value = metadata.get(spec.metadata_key)
if isinstance(value, str) and value:
- credentials[credential_key] = value
+ credentials[spec.credential_key] = value
return credentials
def _catalog_encrypted_target_credentials(
self,
row: AgentCatalog,
) -> dict[str, str]:
"""Decrypt migrated catalog config for cloud target keys only."""
- target_keys = _CLOUD_PROVIDER_TARGET_KEYS.get(row.model_provider)
- if not target_keys or row.encrypted_config is None:
+ if not _is_cloud_provider(row.model_provider) or row.encrypted_config is None:
return {}
- allowed_keys = {credential_key for _, credential_key in target_keys}
+ target_keys = _CLOUD_PROVIDER_TARGET_KEYS[row.model_provider]
+ allowed_keys = {spec.credential_key for spec in target_keys}Verified clean under |
There was a problem hiding this comment.
1 issue found across 9 files (changes from recent commits).
Prompt for AI agents (unresolved issues)
Check if these issues are valid — if so, understand the root cause of each and fix them. If appropriate, use sub-agents to investigate and fix each issue separately.
<file name="tracecat/mcp/server.py">
<violation number="1" location="tracecat/mcp/server.py:605">
P1: Custom provider validation uses org-secret retrieval that ignores v2 `AgentCustomProvider` rows, so valid custom providers can be rejected as unconfigured.</violation>
</file>
Reply with feedback, questions, or to request a fix. Tag @cubic-dev-ai to re-run a review.
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: c5146432e3
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
c514643 to
75e1f3f
Compare
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: 75e1f3f63f
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
3a6faa7 to
4684104
Compare
There was a problem hiding this comment.
💡 Codex Review
tracecat/tracecat/agent/preset/service.py
Lines 210 to 213 in 4684104
AgentPresetVersionRead now carries catalog_id, but build_version_read() never sets it, so every version API response reports catalog_id as null even for catalog-backed versions. That drops model-binding metadata for clients consuming /agent-presets/.../versions/..., and any client that round-trips a fetched version into an update/create flow can unintentionally lose catalog-backed execution and fall back to legacy provider-secret behavior.
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
Summary by cubic
Cut over to LLM Providers v2 with an org-wide catalog, per-workspace enablement, and a unified model picker. Models now resolve by catalog id across agents, presets, workflows, MCP tools, ranker, and SDK; credentials resolve per catalog row; passthrough URLs preserve version suffixes; legacy workflow history/tokens remain compatible.
New Features
agent_addons.base_urlwhen present.ModelSelection(AgentModelUI) is used across builder actions, presets, chat, MCP tools, and the ranker; SDK/DSL/runtime carrycatalog_idend‑to‑end; preset creation rejects disabled catalog ids.Migration
use_workspace_credentials; legacy workflow history and LLM tokens are tolerated via relaxed schema parsing.model: ModelSelection) instead ofmodel_name/model_provider; enable desired models in Workspace → “AI models”, then update presets to pick catalog‑backed models.agent_addonsentitlement.Written for commit 8726094b5d1ffb94fe98eefb5443133ee8a5213a. Summary will update on new commits. Review in cubic