Skip to content

feat(agents): llm providers v2 migration#2566

Merged
jordan-umusu merged 19 commits intomainfrom
feat/llm-providers-v2-migrations
Apr 28, 2026
Merged

feat(agents): llm providers v2 migration#2566
jordan-umusu merged 19 commits intomainfrom
feat/llm-providers-v2-migrations

Conversation

@jordan-umusu
Copy link
Copy Markdown
Collaborator

@jordan-umusu jordan-umusu commented Apr 24, 2026

Summary by cubic

Backfills orgs into the LLM Providers v2 catalog without decrypting secrets, seeds the platform catalog, and ships a control plane for catalog/custom providers, access, and default model selection. Adds tenant RLS, capability flags, and stricter passthrough to prep broader model management.

  • New Features

    • Control plane for catalog CRUD, custom provider CRUD, and model access (org/workspace), with cursor pagination and audit logs.
    • Default model selection via GET/PUT endpoints using catalog_id.
    • Platform catalog bumped with capability flags (supports_native_streaming, supports_parallel_function_calling) and stricter passthrough validation; tenant RLS enforced; dropped custom-provider discovery status.
  • Migration

    • Copies ciphertext from organization_secret agent-{provider}-credentials to encrypted_config on catalog/custom-provider rows (no decryption).
    • Seeds platform catalog from tracecat/agent/platform_catalog.json.
    • Cloud providers (bedrock, azure_openai, azure_ai, vertex_ai): one org-scoped catalog row (model_name = provider) plus an org-wide access grant.
    • Direct providers (openai, anthropic, gemini): org gets access to all matching platform catalog rows; no org catalog rows.
    • Custom provider: creates an agent_custom_provider and linked catalog row (custom-model-provider/custom) with org-wide access.
    • Backfills catalog_id on presets and versions; prefers platform rows over org rows when both match.
    • Resolves agent_default_model to agent_default_model_catalog_id.
    • Idempotent, expansion-only; downgrade is a no-op.

Written for commit e665586. Summary will update on new commits. Review in cubic

@jordan-umusu jordan-umusu changed the title feat(agents): llm providers v2 migrations feat(agents): llm providers v2 migration Apr 24, 2026
@jordan-umusu jordan-umusu added the migration Database migration label Apr 24, 2026
@jordan-umusu jordan-umusu changed the base branch from main to feat/llm-providers-v2-be April 24, 2026 21:56
@jordan-umusu jordan-umusu marked this pull request as ready for review April 24, 2026 21:56
Comment thread alembic/versions/96470fdcc686_v2_backfill_catalog_and_access.py Fixed
Comment thread alembic/versions/96470fdcc686_v2_backfill_catalog_and_access.py Fixed
Comment thread alembic/versions/96470fdcc686_v2_backfill_catalog_and_access.py Fixed
@zeropath-ai
Copy link
Copy Markdown

zeropath-ai Bot commented Apr 24, 2026

No security or compliance issues detected. Reviewed everything up to e665586.

Security Overview
Detected Code Changes
Change Type Relevant files
Configuration changes ► 96470fdcc686_v2_backfill_catalog_and_access.py
    Add initial database migration for v2 LLM provider backfill
► test_agent_catalog_migration.py
    Add tests for v2 LLM provider migration logic
Enhancement ► 96470fdcc686_v2_backfill_catalog_and_access.py
    Backfill organization secrets to new catalog and custom provider tables
    Seed platform catalog from JSON file
    Link presets and preset versions to catalog rows
    Grant access to direct providers based on catalog rows
    Migrate default model settings to use catalog IDs

Copy link
Copy Markdown
Contributor

@cubic-dev-ai cubic-dev-ai Bot left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

1 issue found across 2 files

Confidence score: 2/5

  • There is a high-confidence, high-severity migration risk in alembic/versions/96470fdcc686_v2_backfill_catalog_and_access.py: the per-org try/except does not handle PostgreSQL transaction-abort behavior after a failed statement.
  • If one org-level conn.execute() fails, subsequent statements in the same transaction can keep failing (InFailedSqlTransaction), which can cause the backfill to stop or leave data changes incomplete.
  • Given the concrete failure mode and likely user-visible migration impact, this is high risk to merge until transaction handling is adjusted (e.g., SAVEPOINT/nested transaction strategy).
  • Pay close attention to alembic/versions/96470fdcc686_v2_backfill_catalog_and_access.py - per-org error handling currently won’t recover after database-level statement failure.
Prompt for AI agents (unresolved issues)

Check if these issues are valid — if so, understand the root cause of each and fix them. If appropriate, use sub-agents to investigate and fix each issue separately.


<file name="alembic/versions/96470fdcc686_v2_backfill_catalog_and_access.py">

<violation number="1" location="alembic/versions/96470fdcc686_v2_backfill_catalog_and_access.py:563">
P1: The per-org `try/except` won't recover from database-level errors without a SAVEPOINT. In PostgreSQL, a failed statement inside a transaction marks it as aborted — all subsequent `conn.execute()` calls will raise `InFailedSqlTransaction`, causing every remaining org and the preset backfill to fail too.

Wrap each org's work in `conn.begin_nested()` (which emits `SAVEPOINT` / `ROLLBACK TO SAVEPOINT`) so the outer transaction stays healthy after an individual org failure.</violation>
</file>

Reply with feedback, questions, or to request a fix. Tag @cubic-dev-ai to re-run a review.

Comment thread alembic/versions/96470fdcc686_v2_backfill_catalog_and_access.py Outdated
@jordan-umusu jordan-umusu force-pushed the feat/llm-providers-v2-migrations branch from 3bd0de9 to 97ed3cd Compare April 24, 2026 22:50
@jordan-umusu
Copy link
Copy Markdown
Collaborator Author

@cubic re-review

@cubic-dev-ai
Copy link
Copy Markdown
Contributor

cubic-dev-ai Bot commented Apr 25, 2026

@cubic re-review

@jordan-umusu I have started the AI code review. It will take a few minutes to complete.

Copy link
Copy Markdown
Contributor

@cubic-dev-ai cubic-dev-ai Bot left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

No issues found across 2 files

Confidence score: 5/5

  • Automated review surfaced no issues in the provided summaries.
  • No files require special attention.

@jordan-umusu jordan-umusu force-pushed the feat/llm-providers-v2-be branch from b0407e1 to c15ed89 Compare April 25, 2026 13:04
@jordan-umusu jordan-umusu force-pushed the feat/llm-providers-v2-migrations branch from 97ed3cd to 3dd2d98 Compare April 25, 2026 13:04
@jordan-umusu jordan-umusu force-pushed the feat/llm-providers-v2-be branch 4 times, most recently from de58209 to cb79da3 Compare April 27, 2026 14:22
@jordan-umusu jordan-umusu force-pushed the feat/llm-providers-v2-migrations branch 2 times, most recently from 3a4f6bc to dc41893 Compare April 27, 2026 14:33
@jordan-umusu jordan-umusu force-pushed the feat/llm-providers-v2-be branch from cb79da3 to 86cdaf2 Compare April 27, 2026 14:36
@jordan-umusu jordan-umusu changed the base branch from feat/llm-providers-v2-be to main April 27, 2026 14:37
Comment thread tracecat/agent/provider/schemas.py
Comment thread tracecat/agent/provider/router.py
@jordan-umusu jordan-umusu force-pushed the feat/llm-providers-v2-migrations branch from dc41893 to 7c0ff5a Compare April 27, 2026 15:17
@jordan-umusu jordan-umusu temporarily deployed to internal-registry-ci April 27, 2026 15:17 — with GitHub Actions Inactive
@jordan-umusu jordan-umusu temporarily deployed to internal-registry-ci April 27, 2026 15:17 — with GitHub Actions Inactive
@jordan-umusu jordan-umusu force-pushed the feat/llm-providers-v2-migrations branch from 7c0ff5a to ac4b173 Compare April 27, 2026 15:29
@jordan-umusu jordan-umusu temporarily deployed to internal-registry-ci April 27, 2026 15:29 — with GitHub Actions Inactive
@jordan-umusu jordan-umusu temporarily deployed to internal-registry-ci April 27, 2026 15:30 — with GitHub Actions Inactive
@jordan-umusu jordan-umusu force-pushed the feat/llm-providers-v2-migrations branch from ac4b173 to 8a089ef Compare April 27, 2026 15:37
@jordan-umusu jordan-umusu temporarily deployed to internal-registry-ci April 27, 2026 15:50 — with GitHub Actions Inactive
@jordan-umusu jordan-umusu temporarily deployed to internal-registry-ci April 27, 2026 15:51 — with GitHub Actions Inactive
Copy link
Copy Markdown
Contributor

@daryllimyt daryllimyt left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The data migration can fail or backfill incorrect data for supported existing database states, notably duplicate secret environments and pre-existing custom providers.

Full review comments:

  • [P1] Filter provider secrets by environment — alembic/versions/96470fdcc686_v2_backfill_catalog_and_access.py:307-310
    When an organization has the same agent credential name in multiple secret environments, this lookup ignores organization_secret.environment, so .one_or_none() raises and blocks the whole migration; if only a non-default environment exists, the migration can also copy credentials/grant access that the agent service would not use because runtime lookups default to the default environment. Add the environment column and filter these provider-secret queries to the default environment.

  • [P1] Do not reuse arbitrary custom providers — alembic/versions/96470fdcc686_v2_backfill_catalog_and_access.py:436-440
    For orgs that already have custom-provider rows when this migration runs, this unscoped lookup either raises MultipleResultsFound if there is more than one provider or attaches the legacy custom-model-provider catalog entry to an unrelated existing provider without copying the legacy ciphertext onto that provider. The backfill needs to find a migration-created/provider-specific row or create a dedicated one for the legacy secret.

Comment thread alembic/versions/96470fdcc686_v2_backfill_catalog_and_access.py
Comment thread alembic/versions/96470fdcc686_v2_backfill_catalog_and_access.py
@daryllimyt daryllimyt self-requested a review April 28, 2026 17:31
@jordan-umusu jordan-umusu force-pushed the feat/llm-providers-v2-migrations branch from b1cead5 to e665586 Compare April 28, 2026 19:09
@jordan-umusu jordan-umusu temporarily deployed to internal-registry-ci April 28, 2026 19:10 — with GitHub Actions Inactive
@jordan-umusu jordan-umusu temporarily deployed to internal-registry-ci April 28, 2026 19:10 — with GitHub Actions Inactive
@jordan-umusu jordan-umusu merged commit ec8bc95 into main Apr 28, 2026
16 checks passed
@jordan-umusu jordan-umusu deleted the feat/llm-providers-v2-migrations branch April 28, 2026 19:17
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

migration Database migration

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants