…on (#1625)

* feat: Implement in-process JWT claims caching with TTL and LRU eviction

* style: ruff format (auto)

* refactor: Update cache initialization to use settings for max size and TTL

* feat: code cleanup

* fix: Improve public key file reading and enhance token scheme handling

* style: ruff autofix (auto)

* fix: Handle user insert races and add test timeout (#1618)

* Handle user insert races and add test timeout

Increase TypeScript integration test timeout to 120s to reduce flakiness during slow CI runs.

Enhance user_service.ensure_user_row IntegrityError handling to explicitly handle concurrent-insert races: detect a (oauth_provider, oauth_subject) race and return the existing row, detect an email_lookup_hash race by looking up the email and returning the concurrent identity when it matches, and handle PK collisions by retrying the insert with a new UUID. Add explanatory comments about which collisions are recoverable and when errors should propagate to the caller.

* style: ruff format (auto)

* Use PEP 604 union for agent config return type

Replace typing.Optional[dict] with PEP 604 union syntax (dict | None) for get_effective_agent_config and remove the now-unused Optional import. This is a pure type-annotation cleanup (no runtime behavior changes); note it requires Python 3.10+ for the `|` union syntax.

---------

Co-authored-by: autofix-ci[bot] <114827586+autofix-ci[bot]@users.noreply.github.com>

* fix: display file names and fix ingestion for onedrive (#1609)

* fixed onedrive not working

* style: ruff format (auto)

* removed sites.read.all from onedrive

* style: ruff format (auto)

* fixed allowed users and groups

* fix lint error

* fixed lint

* fixed mypy lint

* style: ruff format (auto)

---------

Co-authored-by: autofix-ci[bot] <114827586+autofix-ci[bot]@users.noreply.github.com>

* fix: Skip deleted connector files; return orphan IDs (#1592)

* Skip deleted connector files; return orphan IDs

Handle files deleted at source and improve orphan reconciliation.

- Add TaskStatus.SKIPPED and update ConnectorFileProcessor to catch FileNotFoundError/ValueError from connector.get_file_content. When a file is missing, mark the file task as SKIPPED, set a skipped result, update timestamps and counters, and avoid raising so the upload can continue.
- Change reconcile_orphans_for_connector_type to return a list of orphan file IDs ([]) instead of an int, update early returns and docstring to reflect returning the deleted/removed document IDs so callers can exclude them from subsequent sync passes.
- Minor whitespace cleanup in sync_all_connectors.

These changes avoid aborting syncs on missing source files and make orphan deletions explicit for downstream processing.

* Skip deleted connector files; return orphan IDs

Handle files deleted at source and improve orphan reconciliation.

- Add TaskStatus.SKIPPED and update ConnectorFileProcessor to catch FileNotFoundError/ValueError from connector.get_file_content. When a file is missing, mark the file task as SKIPPED, set a skipped result, update timestamps and counters, and avoid raising so the upload can continue.
- Change reconcile_orphans_for_connector_type to return a list of orphan file IDs ([]) instead of an int, update early returns and docstring to reflect returning the deleted/removed document IDs so callers can exclude them from subsequent sync passes.
- Minor whitespace cleanup in sync_all_connectors.

These changes avoid aborting syncs on missing source files and make orphan deletions explicit for downstream processing.

* Add *.db to .gitignore

Add a '*.db' pattern to .gitignore to prevent local database files (e.g., SQLite) from being committed to the repository.

* Add orphan reconcile and bulk-delete helper

Introduce a reconcile pass and supporting bulk-delete utility to remove OpenSearch chunks for files that no longer exist remotely.

- Add reconcile_orphans_for_connector_type (src/api/connectors.py): lists all active connections for a connector type, strictly gates on unauthenticated connections or listing errors (abort with 0 deletes), aggregates paginated remote file IDs, computes orphans (indexed IDs not present remotely) and invokes delete_chunks_by_document_ids. Integrated into connector_sync and sync_all_connectors (skips reconcile when sync is capped).
- Add delete_chunks_by_document_ids (src/api/documents.py): issues a single delete_by_query with terms(document_id, ...) and conflicts="proceed", returns deleted count, and short-circuits on empty input.
- Update connector metadata painless params (src/connectors/service.py): include filename param and assign ctx._source.filename = params.filename in the update script so renamed files update indexed chunks.
- Add unit tests: cover delete_chunks_by_document_ids, reconcile_orphans_for_connector_type (gating, pagination, multi-connection union, error handling), and connector metadata filename behavior.

Behavior notes: reconcile is conservative to avoid false-positive deletions; bulk delete is defensive and returns 0 on unexpected responses or failures.

* fix lint

* style: ruff format (auto)

* fix lint

* Use max_files param and add typing

Add an explicit type annotation for files_to_process and update the connector.list_files call to use the max_files parameter instead of limit to match the connector API. Also remove redundant `# type: ignore` comments on connector.cfg assignments as they are no longer necessary. Minor cleanup to improve type clarity and API consistency.

* Add connector sync preview UI and API

Introduce sync-preview functionality for connectors and sync-all flows. Backend: add ID→filename aggregation, compute_orphans (non-destructive orphan detection), orphan deletion helper, and preview endpoints (connector_sync_preview, connectors_sync_all_preview) plus wiring in internal routes. Frontend: add useSyncConnectorPreview/useSyncAllConnectorsPreview mutations, wire preview flows into Knowledge pages/components, and add SyncConfirmDialog component to show orphan lists, per-connector synced counts, availability flags, and confirm deletion+sync interactions. Also update UI buttons to open the preview dialog and handle loading/syncing states.

* style: ruff format (auto)

* Hoist and consolidate imports in processors

Move many inline imports to module scope and consolidate repeated imports across TaskProcessor and its subclasses. Adds top-level imports (asyncio, datetime, mimetypes, os, time), config/settings and utility functions (clients, get_embedding_model, get_index_name, get_openrag_config, extract_relevant, process_text_file, resplit_chunks_character_windows, ensure_embedding_field_exists, auto_cleanup_tempfile, hash_id, build_filename_delete_body, build_filename_search_body) and imports TaskStatus. Removes redundant per-function imports to improve readability and reduce repeated import overhead; no functional changes intended.

* Type UseQueryOptions with SearchResult

Specify UseQueryOptions<SearchResult> for the useGetSearchQuery hook's options parameter to improve TypeScript type inference and ensure the query options expect SearchResult data. This change tightens typings without altering runtime behavior.

---------

Co-authored-by: autofix-ci[bot] <114827586+autofix-ci[bot]@users.noreply.github.com>

* fix: add reset db on command, expose error on e2e tests (#1627)

* fix factory reset not deleting data

* make onboarding errors pop up on e2e tests

* style: ruff format (auto)

* added check for error when uploading

* added check if its on first step to rollback, not only if its complete

* reset correctly

* update timeout for uploading document

* remove misclick

* fixed lint

* fix mypy errors

* style: ruff format (auto)

---------

Co-authored-by: autofix-ci[bot] <114827586+autofix-ci[bot]@users.noreply.github.com>

* refactor: handler for file upload for context (#1624)

* file upload for context

* coderabbit suggestions

* chore:  add operator make commands (#1628)

* Add kind build/load targets and kind sample

Add Makefile targets to build and load app images into a kind cluster (KIND_CLUSTER_NAME default: openrag): kind-load-app-images and kind-build-load-apps, and expose them in the help output. Update kubernetes/operator/README.md with instructions to run the operator locally, apply a kind-local sample, build/load images into kind, and restart deployments after rebuilding. Add a new config sample openrag_v1alpha1_openrag-kind-local.yaml tuned for 2-CPU kind/Colima clusters (lower resource requests, imagePullPolicy: Never). Also annotate the default sample to point users to the kind-local file for local clusters.

* Add Makefile help for operator & kind

Add a new help_operator target and OPERATOR_DIR variable to the Makefile, and include it in the .PHONY list. The new help output documents Kubernetes operator and kind-related commands (build/load app images into kind, operator deps/install/run/build/test/lint/manifests/generate, deploy/undeploy, docker-build, helm install) and provides a typical kind + local images workflow and sample CR usage. This makes it easier for developers to discover and run local operator/cluster tasks from the repo root.

* redirect on logout (#1440) (#1636)

* fix: Copy flows directory into Docker image (#1632)

Add a COPY instruction to include the local flows/ directory at /app/flows in the image so bundled flows are available at runtime. Placed before the entrypoint/ownership handling to ensure the files are copied with the intended UID/permissions.

* feat: settings saas tabs menu (#1637)

* remove logic for previous tab selected, need to be discussed with Ana

* reduce repeated code and use darkmodelight class

* reduce repeated code and use darkmodelight class

* remove unecessary !important

* remove unecessary !important

* remove duplicate css blocks

* remove !important and merge active and hover state

* put back TabsContent anf fix hover issue

---------

Co-authored-by: Olfa Maslah <[email protected]>

* chore: Add ruff autofix step to CI workflows (#1640)

* Add ruff autofix step to CI workflows

Add an in-place Ruff fix step to .github/workflows/autofix.ci.yml that runs uv run ruff check --fix on changed files, rename the job and autofix commit message to reflect "ruff autofix", and update comments to clarify that autofix.ci applies safe fixes and formatting. Also update .github/workflows/lint-backend.yml comments to state that safe lint fixes and formatting are auto-applied by autofix.ci while keeping strict lint (--no-fix) and mypy in the lint workflow.

* Update autofix.ci.yml

* fix: Enum IDs and delete OpenSearch docs by _id (#1638)

* Enum IDs and delete OpenSearch docs by _id

Add DLS-safe OpenSearch delete helpers and use them to avoid silent no-ops from delete_by_query. Introduces utils/opensearch_delete.py with collect_visible_document_ids and delete_document_ids (enumerate visible _ids via search/scroll, then delete by primary _id). Update delete_chunks_by_document_ids to enumerate chunk IDs and delete each by _id. Ensure langflow_connector_service and TaskProcessor clear stale chunks (by document_id) before re-ingest/re-index to prevent duplicate or trailing chunks after renames. Improve connector listing by scoping cfg.file_ids/folder_ids when available to avoid false orphan detection. Add and update unit tests to assert the new enumeration-and-delete behavior and ordering.

* style: ruff format (auto)

* ruff fix

* ruff fix

---------

Co-authored-by: autofix-ci[bot] <114827586+autofix-ci[bot]@users.noreply.github.com>

* make probes more robust (#1642)

* feat: Show google_drive connector for cloud brand (#1650)

Remove google_drive from the cloud-brand exclusion so Google Drive connectors are visible when isCloudBrand is true. Applied the change in connector-cards.tsx and knowledge-dropdown.tsx (OneDrive remains excluded).

* operator ubi base build (#1652)

* remove flow download initContainer when flowRef is removed (#1653)

* fix: cr deletion (#1656)

* fix cr deletion

* refactoring

* fix unit test

* fix lint

* fix cross namespace CR deletion

* Potential fix for pull request finding 'CodeQL / Information exposure through an exception'

Co-authored-by: Copilot Autofix powered by AI <62310815+github-advanced-security[bot]@users.noreply.github.com>

---------

Co-authored-by: autofix-ci[bot] <114827586+autofix-ci[bot]@users.noreply.github.com>
Co-authored-by: Edwin Jose <[email protected]>
Co-authored-by: Lucas Oliveira <[email protected]>
Co-authored-by: Mike Fortman <[email protected]>
Co-authored-by: Wallgau <[email protected]>
Co-authored-by: Olfa Maslah <[email protected]>
Co-authored-by: ming <[email protected]>
Co-authored-by: Copilot Autofix powered by AI <62310815+github-advanced-security[bot]@users.noreply.github.com>

…on (#1625)

* feat: Implement in-process JWT claims caching with TTL and LRU eviction

* style: ruff format (auto)

* refactor: Update cache initialization to use settings for max size and TTL

* feat: code cleanup

* fix: Improve public key file reading and enhance token scheme handling

* style: ruff autofix (auto)

* fix: Handle user insert races and add test timeout (#1618)

* Handle user insert races and add test timeout

Increase TypeScript integration test timeout to 120s to reduce flakiness during slow CI runs.

Enhance user_service.ensure_user_row IntegrityError handling to explicitly handle concurrent-insert races: detect a (oauth_provider, oauth_subject) race and return the existing row, detect an email_lookup_hash race by looking up the email and returning the concurrent identity when it matches, and handle PK collisions by retrying the insert with a new UUID. Add explanatory comments about which collisions are recoverable and when errors should propagate to the caller.

* style: ruff format (auto)

* Use PEP 604 union for agent config return type

Replace typing.Optional[dict] with PEP 604 union syntax (dict | None) for get_effective_agent_config and remove the now-unused Optional import. This is a pure type-annotation cleanup (no runtime behavior changes); note it requires Python 3.10+ for the `|` union syntax.

---------

Co-authored-by: autofix-ci[bot] <114827586+autofix-ci[bot]@users.noreply.github.com>

* fix: display file names and fix ingestion for onedrive (#1609)

* fixed onedrive not working

* style: ruff format (auto)

* removed sites.read.all from onedrive

* style: ruff format (auto)

* fixed allowed users and groups

* fix lint error

* fixed lint

* fixed mypy lint

* style: ruff format (auto)

---------

Co-authored-by: autofix-ci[bot] <114827586+autofix-ci[bot]@users.noreply.github.com>

* fix: Skip deleted connector files; return orphan IDs (#1592)

* Skip deleted connector files; return orphan IDs

Handle files deleted at source and improve orphan reconciliation.

- Add TaskStatus.SKIPPED and update ConnectorFileProcessor to catch FileNotFoundError/ValueError from connector.get_file_content. When a file is missing, mark the file task as SKIPPED, set a skipped result, update timestamps and counters, and avoid raising so the upload can continue.
- Change reconcile_orphans_for_connector_type to return a list of orphan file IDs ([]) instead of an int, update early returns and docstring to reflect returning the deleted/removed document IDs so callers can exclude them from subsequent sync passes.
- Minor whitespace cleanup in sync_all_connectors.

These changes avoid aborting syncs on missing source files and make orphan deletions explicit for downstream processing.

* Skip deleted connector files; return orphan IDs

Handle files deleted at source and improve orphan reconciliation.

- Add TaskStatus.SKIPPED and update ConnectorFileProcessor to catch FileNotFoundError/ValueError from connector.get_file_content. When a file is missing, mark the file task as SKIPPED, set a skipped result, update timestamps and counters, and avoid raising so the upload can continue.
- Change reconcile_orphans_for_connector_type to return a list of orphan file IDs ([]) instead of an int, update early returns and docstring to reflect returning the deleted/removed document IDs so callers can exclude them from subsequent sync passes.
- Minor whitespace cleanup in sync_all_connectors.

These changes avoid aborting syncs on missing source files and make orphan deletions explicit for downstream processing.

* Add *.db to .gitignore

Add a '*.db' pattern to .gitignore to prevent local database files (e.g., SQLite) from being committed to the repository.

* Add orphan reconcile and bulk-delete helper

Introduce a reconcile pass and supporting bulk-delete utility to remove OpenSearch chunks for files that no longer exist remotely.

- Add reconcile_orphans_for_connector_type (src/api/connectors.py): lists all active connections for a connector type, strictly gates on unauthenticated connections or listing errors (abort with 0 deletes), aggregates paginated remote file IDs, computes orphans (indexed IDs not present remotely) and invokes delete_chunks_by_document_ids. Integrated into connector_sync and sync_all_connectors (skips reconcile when sync is capped).
- Add delete_chunks_by_document_ids (src/api/documents.py): issues a single delete_by_query with terms(document_id, ...) and conflicts="proceed", returns deleted count, and short-circuits on empty input.
- Update connector metadata painless params (src/connectors/service.py): include filename param and assign ctx._source.filename = params.filename in the update script so renamed files update indexed chunks.
- Add unit tests: cover delete_chunks_by_document_ids, reconcile_orphans_for_connector_type (gating, pagination, multi-connection union, error handling), and connector metadata filename behavior.

Behavior notes: reconcile is conservative to avoid false-positive deletions; bulk delete is defensive and returns 0 on unexpected responses or failures.

* fix lint

* style: ruff format (auto)

* fix lint

* Use max_files param and add typing

Add an explicit type annotation for files_to_process and update the connector.list_files call to use the max_files parameter instead of limit to match the connector API. Also remove redundant `# type: ignore` comments on connector.cfg assignments as they are no longer necessary. Minor cleanup to improve type clarity and API consistency.

* Add connector sync preview UI and API

Introduce sync-preview functionality for connectors and sync-all flows. Backend: add ID→filename aggregation, compute_orphans (non-destructive orphan detection), orphan deletion helper, and preview endpoints (connector_sync_preview, connectors_sync_all_preview) plus wiring in internal routes. Frontend: add useSyncConnectorPreview/useSyncAllConnectorsPreview mutations, wire preview flows into Knowledge pages/components, and add SyncConfirmDialog component to show orphan lists, per-connector synced counts, availability flags, and confirm deletion+sync interactions. Also update UI buttons to open the preview dialog and handle loading/syncing states.

* style: ruff format (auto)

* Hoist and consolidate imports in processors

Move many inline imports to module scope and consolidate repeated imports across TaskProcessor and its subclasses. Adds top-level imports (asyncio, datetime, mimetypes, os, time), config/settings and utility functions (clients, get_embedding_model, get_index_name, get_openrag_config, extract_relevant, process_text_file, resplit_chunks_character_windows, ensure_embedding_field_exists, auto_cleanup_tempfile, hash_id, build_filename_delete_body, build_filename_search_body) and imports TaskStatus. Removes redundant per-function imports to improve readability and reduce repeated import overhead; no functional changes intended.

* Type UseQueryOptions with SearchResult

Specify UseQueryOptions<SearchResult> for the useGetSearchQuery hook's options parameter to improve TypeScript type inference and ensure the query options expect SearchResult data. This change tightens typings without altering runtime behavior.

---------

Co-authored-by: autofix-ci[bot] <114827586+autofix-ci[bot]@users.noreply.github.com>

* fix: add reset db on command, expose error on e2e tests (#1627)

* fix factory reset not deleting data

* make onboarding errors pop up on e2e tests

* style: ruff format (auto)

* added check for error when uploading

* added check if its on first step to rollback, not only if its complete

* reset correctly

* update timeout for uploading document

* remove misclick

* fixed lint

* fix mypy errors

* style: ruff format (auto)

---------

Co-authored-by: autofix-ci[bot] <114827586+autofix-ci[bot]@users.noreply.github.com>

* refactor: handler for file upload for context (#1624)

* file upload for context

* coderabbit suggestions

* chore:  add operator make commands (#1628)

* Add kind build/load targets and kind sample

Add Makefile targets to build and load app images into a kind cluster (KIND_CLUSTER_NAME default: openrag): kind-load-app-images and kind-build-load-apps, and expose them in the help output. Update kubernetes/operator/README.md with instructions to run the operator locally, apply a kind-local sample, build/load images into kind, and restart deployments after rebuilding. Add a new config sample openrag_v1alpha1_openrag-kind-local.yaml tuned for 2-CPU kind/Colima clusters (lower resource requests, imagePullPolicy: Never). Also annotate the default sample to point users to the kind-local file for local clusters.

* Add Makefile help for operator & kind

Add a new help_operator target and OPERATOR_DIR variable to the Makefile, and include it in the .PHONY list. The new help output documents Kubernetes operator and kind-related commands (build/load app images into kind, operator deps/install/run/build/test/lint/manifests/generate, deploy/undeploy, docker-build, helm install) and provides a typical kind + local images workflow and sample CR usage. This makes it easier for developers to discover and run local operator/cluster tasks from the repo root.

* redirect on logout (#1440) (#1636)

* fix: Copy flows directory into Docker image (#1632)

Add a COPY instruction to include the local flows/ directory at /app/flows in the image so bundled flows are available at runtime. Placed before the entrypoint/ownership handling to ensure the files are copied with the intended UID/permissions.

* feat: settings saas tabs menu (#1637)

* remove logic for previous tab selected, need to be discussed with Ana

* reduce repeated code and use darkmodelight class

* reduce repeated code and use darkmodelight class

* remove unecessary !important

* remove unecessary !important

* remove duplicate css blocks

* remove !important and merge active and hover state

* put back TabsContent anf fix hover issue

---------

Co-authored-by: Olfa Maslah <[email protected]>

* chore: Add ruff autofix step to CI workflows (#1640)

* Add ruff autofix step to CI workflows

Add an in-place Ruff fix step to .github/workflows/autofix.ci.yml that runs uv run ruff check --fix on changed files, rename the job and autofix commit message to reflect "ruff autofix", and update comments to clarify that autofix.ci applies safe fixes and formatting. Also update .github/workflows/lint-backend.yml comments to state that safe lint fixes and formatting are auto-applied by autofix.ci while keeping strict lint (--no-fix) and mypy in the lint workflow.

* Update autofix.ci.yml

* fix: Enum IDs and delete OpenSearch docs by _id (#1638)

* Enum IDs and delete OpenSearch docs by _id

Add DLS-safe OpenSearch delete helpers and use them to avoid silent no-ops from delete_by_query. Introduces utils/opensearch_delete.py with collect_visible_document_ids and delete_document_ids (enumerate visible _ids via search/scroll, then delete by primary _id). Update delete_chunks_by_document_ids to enumerate chunk IDs and delete each by _id. Ensure langflow_connector_service and TaskProcessor clear stale chunks (by document_id) before re-ingest/re-index to prevent duplicate or trailing chunks after renames. Improve connector listing by scoping cfg.file_ids/folder_ids when available to avoid false orphan detection. Add and update unit tests to assert the new enumeration-and-delete behavior and ordering.

* style: ruff format (auto)

* ruff fix

* ruff fix

---------

Co-authored-by: autofix-ci[bot] <114827586+autofix-ci[bot]@users.noreply.github.com>

* make probes more robust (#1642)

* feat: Show google_drive connector for cloud brand (#1650)

Remove google_drive from the cloud-brand exclusion so Google Drive connectors are visible when isCloudBrand is true. Applied the change in connector-cards.tsx and knowledge-dropdown.tsx (OneDrive remains excluded).

* operator ubi base build (#1652)

* remove flow download initContainer when flowRef is removed (#1653)

* fix: cr deletion (#1656)

* fix cr deletion

* refactoring

* fix unit test

* fix lint

* fix cross namespace CR deletion

* Potential fix for pull request finding 'CodeQL / Information exposure through an exception'

Co-authored-by: Copilot Autofix powered by AI <62310815+github-advanced-security[bot]@users.noreply.github.com>

---------

Co-authored-by: autofix-ci[bot] <114827586+autofix-ci[bot]@users.noreply.github.com>
Co-authored-by: Edwin Jose <[email protected]>
Co-authored-by: Lucas Oliveira <[email protected]>
Co-authored-by: Mike Fortman <[email protected]>
Co-authored-by: Wallgau <[email protected]>
Co-authored-by: Olfa Maslah <[email protected]>
Co-authored-by: ming <[email protected]>
Co-authored-by: Copilot Autofix powered by AI <62310815+github-advanced-security[bot]@users.noreply.github.com>