Skip to content

Add/model dataset group#1481

Merged
mina-parham merged 103 commits intomainfrom
add/model-dataset-group
Mar 23, 2026
Merged

Add/model dataset group#1481
mina-parham merged 103 commits intomainfrom
add/model-dataset-group

Conversation

@mina-parham
Copy link
Copy Markdown
Contributor

@mina-parham mina-parham commented Mar 6, 2026

  • Save a model to registry:
  1. Run a fine-tuning job to completion
  2. Use the "Save to Registry" action on the finished job
  3. Confirm the response now includes a `version` field in the JSON
  • Save a dataset to registry:
  1. Run a data-generating job to completion
  2. Use "Save to Registry" on the dataset
  3. Confirm the response includes a `version` field
  • Navigate to the Models page
    • Each model that has version entries should show a VersionGroupChip badge
    • Click the chip to open the AssetVersionsDrawer and verify all versions are listed with their version number, tag, job ID, and creation date
  1. Navigate to the Datasets page
    • Same as above — dataset cards should show version group chips where applicable
    • Click to expand the drawer and inspect version details

@mina-parham mina-parham marked this pull request as draft March 6, 2026 21:12
@paragon-review
Copy link
Copy Markdown

paragon-review bot commented Mar 6, 2026

Paragon Summary

This pull request review identified 17 issues across 3 categories in 16 files. The review analyzed code changes, potential bugs, security vulnerabilities, performance issues, and code quality concerns using automated analysis tools.

This PR adds version grouping functionality for models and datasets, introducing an asset versions database table, backend service/API endpoints, and frontend UI components (VersionGroupChip and AssetVersionsDrawer) to display and manage grouped asset versions.

Key changes:

  • Asset versioning system: New DB table, service, router, and UI drawer for managing versions
  • Version grouping UI: Added VersionGroupChip and AssetVersionsDrawer components
  • Model & dataset integration: Updated routers and components to support versioned assets
  • API endpoints: Exposed new asset version endpoints in frontend client
  • Database migration: Created asset_versions table via Alembic

Confidence score: 3/5

  • This PR has moderate risk due to 5 high-priority issues that should be addressed
  • Score reflects significant bugs, performance issues, or architectural concerns
  • Review high-priority findings carefully before merging

16 files reviewed, 17 comments

Severity breakdown: High: 5, Medium: 8, Low: 4


Tip: @paragon-run <instructions> to chat with our agent or push fixes!

Dashboard

Comment thread api/transformerlab/services/asset_version_service.py Outdated
Endpoints.AssetVersions = {
ListGroups: (assetType: string) =>
`${API_URL()}asset_versions/groups?asset_type=${assetType}`,
DeleteGroup: (assetType: string, groupName: string) =>
Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Bug: No input sanitization on path-segment parameters groupName/assetType

No input sanitization on path-segment parameters groupName/assetType. Slashes or encoded chars could cause routing issues. Add encodeURIComponent in endpoint builders.

View Details

Location: src/renderer/lib/api-client/endpoints.ts (lines 557)

Analysis

No input sanitization on path-segment parameters groupName/assetType

What fails groupName and assetType are interpolated directly into URL path segments without encoding. Group names containing slashes, spaces, or special chars will break routing or hit wrong endpoints.
Result URL path is malformed. The slash in the group name creates extra path segments, causing 404 or routing to unintended endpoints.
Expected All path parameters should be wrapped in encodeURIComponent() to safely handle special characters.
Impact Broken API calls for any group name with special characters. Could also enable path traversal if the backend doesn't validate.
How to reproduce
Create a version group with name 'my model/v2'. Then call ListVersions('model', 'my model/v2'). The URL becomes `.../versions/model/my model/v2` which routes to the wrong endpoint.
Patch Details
-  ListVersions: (assetType: string, groupName: string) =>
-    `${API_URL()}asset_versions/versions/${assetType}/${groupName}`,
+  ListVersions: (assetType: string, groupName: string) =>
+    `${API_URL()}asset_versions/versions/${encodeURIComponent(assetType)}/${encodeURIComponent(groupName)}`,
AI Fix Prompt
Fix this issue: No input sanitization on path-segment parameters groupName/assetType. Slashes or encoded chars could cause routing issues. Add encodeURIComponent in endpoint builders.

Location: src/renderer/lib/api-client/endpoints.ts (lines 557)
Problem: groupName and assetType are interpolated directly into URL path segments without encoding. Group names containing slashes, spaces, or special chars will break routing or hit wrong endpoints.
Current behavior: URL path is malformed. The slash in the group name creates extra path segments, causing 404 or routing to unintended endpoints.
Expected: All path parameters should be wrapped in encodeURIComponent() to safely handle special characters.
Steps to reproduce: Create a version group with name 'my model/v2'. Then call ListVersions('model', 'my model/v2'). The URL becomes `.../versions/model/my model/v2` which routes to the wrong endpoint.

Provide a code fix.


Tip: Reply with @paragon-run to automatically fix this issue

connection = op.get_bind()

# Helper function to check if table exists
def table_exists(table_name: str) -> bool:
Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Bug: Migration uses sqlite_master, breaking Postgres

Migration uses sqlite_master, breaking Postgres. AGENTS.md says both SQLite and Postgres are supported. Use dialect-agnostic table existence check via sa.inspect.

View Details

Location: api/alembic/versions/a3d2e5f8c901_create_asset_versions_table.py (lines 27)

Analysis

Migration uses sqlite_master, breaking Postgres. AGENTS

What fails The Alembic migration hard-codes a SQLite-only system table query (sqlite_master) to check if the table exists, which fails on Postgres deployments.
Result Migration fails with a SQL error on Postgres: relation 'sqlite_master' does not exist.
Expected Migration should run successfully on both SQLite and Postgres using dialect-agnostic introspection.
Impact Blocks deployment on Postgres environments. AGENTS.md explicitly states the app supports both SQLite and Postgres.
How to reproduce
Run `alembic upgrade head` against a PostgreSQL database. The query `SELECT name FROM sqlite_master WHERE type='table' AND name=:name` will raise an error because sqlite_master doesn't exist in Postgres.
Patch Details
-    def table_exists(table_name: str) -> bool:
-        result = connection.execute(
-            sa.text("SELECT name FROM sqlite_master WHERE type='table' AND name=:name"), {"name": table_name}
-        )
-        return result.fetchone() is not None
+    def table_exists(table_name: str) -> bool:
+        return sa.inspect(connection).has_table(table_name)
AI Fix Prompt
Fix this issue: Migration uses sqlite_master, breaking Postgres. AGENTS.md says both SQLite and Postgres are supported. Use dialect-agnostic table existence check via sa.inspect.

Location: api/alembic/versions/a3d2e5f8c901_create_asset_versions_table.py (lines 27)
Problem: The Alembic migration hard-codes a SQLite-only system table query (sqlite_master) to check if the table exists, which fails on Postgres deployments.
Current behavior: Migration fails with a SQL error on Postgres: relation 'sqlite_master' does not exist.
Expected: Migration should run successfully on both SQLite and Postgres using dialect-agnostic introspection.
Steps to reproduce: Run `alembic upgrade head` against a PostgreSQL database. The query `SELECT name FROM sqlite_master WHERE type='table' AND name=:name` will raise an error because sqlite_master doesn't exist in Postgres.

Provide a code fix.


Tip: Reply with @paragon-run to automatically fix this issue

Comment thread api/transformerlab/routers/experiment/jobs.py
Comment thread api/transformerlab/routers/experiment/jobs.py Outdated
version?: number,
) => {
let url = `${API_URL()}asset_versions/resolve/${assetType}/${groupName}`;
const params: string[] = [];
Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Bug: Endpoint URL params not encoded in Resolve builder

Endpoint URL params not encoded in Resolve builder. Tag/version values with special chars corrupt query string. Use encodeURIComponent on param values.

View Details

Location: src/renderer/lib/api-client/endpoints.ts (lines 577)

Analysis

Endpoint URL params not encoded in Resolve builder. Tag/version values with special chars corrupt qu

What fails The Resolve endpoint builder interpolates tag and version values directly into query string parameters without encoding. Tags containing '&' or '=' could corrupt the URL.
Result Query string corruption: extra parameters injected into the URL, potentially causing wrong behavior on the backend.
Expected Use encodeURIComponent() for tag and version values: tag=${encodeURIComponent(tag)}.
Impact Broken API calls for edge-case tag values. Minor query-string injection risk.
How to reproduce
Call Endpoints.AssetVersions.Resolve('model', 'group', 'tag=evil&admin=true'). The resulting URL has injected query parameters.
Patch Details
-    if (tag) params.push(`tag=${tag}`);
-    if (version !== undefined) params.push(`version=${version}`);
+    if (tag) params.push(`tag=${encodeURIComponent(tag)}`);
+    if (version !== undefined) params.push(`version=${encodeURIComponent(String(version))}`);
AI Fix Prompt
Fix this issue: Endpoint URL params not encoded in Resolve builder. Tag/version values with special chars corrupt query string. Use encodeURIComponent on param values.

Location: src/renderer/lib/api-client/endpoints.ts (lines 577)
Problem: The Resolve endpoint builder interpolates tag and version values directly into query string parameters without encoding. Tags containing '&' or '=' could corrupt the URL.
Current behavior: Query string corruption: extra parameters injected into the URL, potentially causing wrong behavior on the backend.
Expected: Use encodeURIComponent() for tag and version values: `tag=${encodeURIComponent(tag)}`.
Steps to reproduce: Call Endpoints.AssetVersions.Resolve('model', 'group', 'tag=evil&admin=true'). The resulting URL has injected query parameters.

Provide a code fix.


Tip: Reply with @paragon-run to automatically fix this issue

@@ -33,6 +35,7 @@ export default function DatasetCard({
parentMutate,
Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Bug: DatasetCard versionGroups prop is untyped (no TypeScript interface)

DatasetCard versionGroups prop is untyped (no TypeScript interface). Props are all untyped in this component. Add explicit prop types for versionGroups.

View Details

Location: src/renderer/components/Data/DatasetCard.tsx (lines 35)

Analysis

DatasetCard versionGroups prop is untyped (no TypeScript interface)

What fails The DatasetCard component destructures props without TypeScript interface. The new versionGroups prop defaults to [] but has no type annotation, violating AGENTS.md strict typing rule.
Result No compile-time type checking on versionGroups prop. Passing wrong shape data won't be caught at build time.
Expected AGENTS.md says 'Avoid any. Define interfaces for all props and API responses to ensure type safety.' A Props interface should be defined.
Impact Reduced type safety. Runtime errors possible if wrong data shape is passed to versionGroups.
How to reproduce
Review DatasetCard.tsx:35-40. All props including the new versionGroups are untyped destructured parameters.
AI Fix Prompt
Fix this issue: DatasetCard versionGroups prop is untyped (no TypeScript interface). Props are all untyped in this component. Add explicit prop types for versionGroups.

Location: src/renderer/components/Data/DatasetCard.tsx (lines 35)
Problem: The DatasetCard component destructures props without TypeScript interface. The new versionGroups prop defaults to [] but has no type annotation, violating AGENTS.md strict typing rule.
Current behavior: No compile-time type checking on versionGroups prop. Passing wrong shape data won't be caught at build time.
Expected: AGENTS.md says 'Avoid any. Define interfaces for all props and API responses to ensure type safety.' A Props interface should be defined.
Steps to reproduce: Review DatasetCard.tsx:35-40. All props including the new versionGroups are untyped destructured parameters.

Provide a code fix.


Tip: Reply with @paragon-run to automatically fix this issue

description: Mapped[Optional[str]] = mapped_column(String, nullable=True)
created_at: Mapped[DateTime] = mapped_column(DateTime, server_default=func.now(), nullable=False)

__table_args__ = (
Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Bug: Duplicate index definitions between ORM model and migration

Duplicate index definitions between ORM model and migration. Same indexes defined in __table_args__ and migration. Remove from one location to avoid conflicts.

View Details

Location: api/transformerlab/shared/models/models.py (lines 314)

Analysis

Duplicate index definitions between ORM model and migration

What fails Index 'idx_asset_versions_group', 'idx_asset_versions_tag', and 'idx_asset_versions_asset_id' are defined in both the Alembic migration and the ORM __table_args__. Additionally, the migration creates extra single-column indexes (ix_asset_versions_asset_type, etc.) that overlap with columns already having index=True in the ORM model.
Result Redundant index definitions that could cause conflicts during autogeneration of future migrations, or errors if Alembic tries to create indexes that already exist.
Expected Indexes should be defined in one authoritative place (preferably the ORM model) and the migration should be auto-generated from it.
Impact Maintenance burden: index changes must be synchronized in two places. Risk of migration errors on fresh databases.
How to reproduce
Compare indexes in the migration (lines 47-58) with __table_args__ in models.py (lines 314-318) and column-level index=True attributes (lines 301-308).
AI Fix Prompt
Fix this issue: Duplicate index definitions between ORM model and migration. Same indexes defined in __table_args__ and migration. Remove from one location to avoid conflicts.

Location: api/transformerlab/shared/models/models.py (lines 314)
Problem: Index 'idx_asset_versions_group', 'idx_asset_versions_tag', and 'idx_asset_versions_asset_id' are defined in both the Alembic migration and the ORM __table_args__. Additionally, the migration creates extra single-column indexes (ix_asset_versions_asset_type, etc.) that overlap with columns already having index=True in the ORM model.
Current behavior: Redundant index definitions that could cause conflicts during autogeneration of future migrations, or errors if Alembic tries to create indexes that already exist.
Expected: Indexes should be defined in one authoritative place (preferably the ORM model) and the migration should be auto-generated from it.
Steps to reproduce: Compare indexes in the migration (lines 47-58) with __table_args__ in models.py (lines 314-318) and column-level index=True attributes (lines 301-308).

Provide a code fix.


Tip: Reply with @paragon-run to automatically fix this issue

model["version_groups"] = group_map[model_id]
else:
model["version_groups"] = []
except Exception as e:
Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Bug: Silent failure on version group augmentation in model and data list

Silent failure on version group augmentation in model and data list. Bare except catches all errors and only prints. Add proper logging instead of print.

View Details

Location: api/transformerlab/routers/model.py (lines 833)

Analysis

Silent failure on version group augmentation in model and data list

What fails Both model_local_list and dataset_list catch all exceptions from version group augmentation with a broad except Exception and only print a warning. Database connection failures, import errors, or schema mismatches are silently swallowed.
Result Version group data silently missing from responses. Only a print statement indicates the failure. No structured logging for monitoring/alerting.
Expected Use the application logger (not print) at WARNING level, and consider whether some errors (like DB connection failures) should propagate.
Impact Debugging difficulty: silent failures in production with no structured logging. Missing version data without any user-visible indication.
How to reproduce
Break the asset_versions table (e.g., drop it). Call GET /model/list. The warning is printed to stdout but no error is visible to API callers or monitoring.
AI Fix Prompt
Fix this issue: Silent failure on version group augmentation in model and data list. Bare except catches all errors and only prints. Add proper logging instead of print.

Location: api/transformerlab/routers/model.py (lines 833)
Problem: Both model_local_list and dataset_list catch all exceptions from version group augmentation with a broad `except Exception` and only print a warning. Database connection failures, import errors, or schema mismatches are silently swallowed.
Current behavior: Version group data silently missing from responses. Only a print statement indicates the failure. No structured logging for monitoring/alerting.
Expected: Use the application logger (not print) at WARNING level, and consider whether some errors (like DB connection failures) should propagate.
Steps to reproduce: Break the asset_versions table (e.g., drop it). Call GET /model/list. The warning is printed to stdout but no error is visible to API callers or monitoring.

Provide a code fix.


Tip: Reply with @paragon-run to automatically fix this issue

Comment thread api/transformerlab/services/asset_version_service.py Outdated
@sentry
Copy link
Copy Markdown

sentry bot commented Mar 9, 2026

@mina-parham mina-parham marked this pull request as ready for review March 9, 2026 19:15
Copy link
Copy Markdown
Member

@deep1401 deep1401 left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I fixed a bug in get_artifacts_from_directory and also fixed the down version for alembic.

When trying to click "Save to Registry" on a completed model in the "View Models" modal, it just gets stuck. I waited for 3 mins but there was no error anywhere.

Could you let me know if this is an issue or if I'm testing it wrong somehow?

Copy link
Copy Markdown
Member

@deep1401 deep1401 left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Keeping aside the model name/group name discrepancy discussion on Discord, I could find 2 bugs. One is that the copy background task that happens seems to create an actual task in the tasks list:

Image

The other bug is that I saved a model and then added another model to the existing group with a different version and it did overwrite my model existing in the models folder:

Image Image

@mina-parham
Copy link
Copy Markdown
Contributor Author

Keeping aside the model name/group name discrepancy discussion on Discord, I could find 2 bugs. One is that the copy background task that happens seems to create an actual task in the tasks list:

Image The other bug is that I saved a model and then added another model to the existing group with a different version and it did overwrite my model existing in the models folder:

Image Image

added both of these changes

Copy link
Copy Markdown
Member

@deep1401 deep1401 left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Image Image

Saving a model to a new group works properly but I think there might be a bug from the previous versions where adding to existing group doesnt work for me.

@dadmobile
Copy link
Copy Markdown
Member

Can you take a look at the merge conflict @mina-parham ? I'm not sure what to do with that.

Comment thread src/renderer/components/Shared/AssetVersionsDrawer.tsx Fixed
@mina-parham
Copy link
Copy Markdown
Contributor Author

Image Image
Saving a model to a new group works properly but I think there might be a bug from the previous versions where adding to existing group doesnt work for me.

now it's working on my side, can you check again

@mina-parham mina-parham merged commit 745cc2d into main Mar 23, 2026
12 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

4 participants