Skip to content

chore: add operator make commands#1628

Merged
edwinjosechittilappilly merged 2 commits into
mainfrom
feat-add-operator-make-commands
May 19, 2026
Merged

chore: add operator make commands#1628
edwinjosechittilappilly merged 2 commits into
mainfrom
feat-add-operator-make-commands

Conversation

@edwinjosechittilappilly

@edwinjosechittilappilly edwinjosechittilappilly commented May 19, 2026

Copy link
Copy Markdown
Collaborator

This pull request adds comprehensive support for local Kubernetes development using kind and Docker/Colima, including new Makefile targets and documentation to streamline building, loading, and deploying OpenRAG app and operator images into a local cluster. It introduces a new sample CR optimized for low-resource environments and clarifies workflows for local testing.

Makefile and CLI improvements:

  • Added new kind-build-load-apps and kind-load-app-images targets to the Makefile for building OpenRAG app images and loading them into a local kind cluster. Also updated help output to document these commands and added a new help_operator target with a detailed operator workflow summary. [1] [2] [3] [4]

Documentation updates:

  • Updated kubernetes/operator/README.md to document the local kind workflow, including building/loading images, applying the new sample CR, and restarting deployments after rebuilding images. Also added instructions for building and loading the operator image for in-cluster use.

Kubernetes operator samples:

  • Added a new sample CR openrag_v1alpha1_openrag-kind-local.yaml with reduced CPU/memory requests and imagePullPolicy: Never for use in local clusters with pre-loaded images.
  • Updated the default sample CR openrag_v1alpha1_openrag.yaml to direct local users to the new kind-specific sample.

Summary by CodeRabbit

  • New Features

    • Added make commands for building and loading application images into local Kubernetes kind clusters
    • Added sample configuration file for deploying OpenRAG on local kind/Colima clusters with optimized settings for low-resource environments
  • Documentation

    • Updated local operator setup instructions with simplified and streamlined workflow

Review Change Stack

Add Makefile targets to build and load app images into a kind cluster (KIND_CLUSTER_NAME default: openrag): kind-load-app-images and kind-build-load-apps, and expose them in the help output. Update kubernetes/operator/README.md with instructions to run the operator locally, apply a kind-local sample, build/load images into kind, and restart deployments after rebuilding. Add a new config sample openrag_v1alpha1_openrag-kind-local.yaml tuned for 2-CPU kind/Colima clusters (lower resource requests, imagePullPolicy: Never). Also annotate the default sample to point users to the kind-local file for local clusters.
Add a new help_operator target and OPERATOR_DIR variable to the Makefile, and include it in the .PHONY list. The new help output documents Kubernetes operator and kind-related commands (build/load app images into kind, operator deps/install/run/build/test/lint/manifests/generate, deploy/undeploy, docker-build, helm install) and provides a typical kind + local images workflow and sample CR usage. This makes it easier for developers to discover and run local operator/cluster tasks from the repo root.
@coderabbitai

coderabbitai Bot commented May 19, 2026

Copy link
Copy Markdown
Contributor

Walkthrough

This PR enables OpenRAG operator development on local kind/Colima clusters by adding make targets to load Docker images, documenting workflows in help text and operator README, and providing a low-resource Kubernetes sample manifest optimized for 2-CPU local environments.

Changes

Local kind/Colima development support

Layer / File(s) Summary
Makefile kind image workflow targets
Makefile
KIND_CLUSTER_NAME variable and kind-load-app-images target verify kind availability and load backend/frontend/langflow images; kind-build-load-apps builds then loads.
Makefile help documentation
Makefile
help_operator PHONY target and section document operator/kind commands, workflows, and kubectl/helm examples; Docker help updated to reference kind targets.
Operator local development instructions
kubernetes/operator/README.md
Local development section replaces go run with make run workflow, includes sample CR application, kind-local CPU notes, image rebuild/reload guidance, and optional operator image build/deploy.
Kubernetes sample manifests for local deployments
kubernetes/operator/config/samples/openrag_v1alpha1_openrag-kind-local.yaml, kubernetes/operator/config/samples/openrag_v1alpha1_openrag.yaml
New kind-local sample YAML with preloaded images (imagePullPolicy: Never), resource requests/limits, storage, and disabled network policies; existing sample updated with comment directing 2-CPU local users to the kind-local variant.

Estimated code review effort

🎯 2 (Simple) | ⏱️ ~12 minutes

Suggested labels

enhancement

🚥 Pre-merge checks | ✅ 5
✅ Passed checks (5 passed)
Check name Status Explanation
Description Check ✅ Passed Check skipped - CodeRabbit’s high-level summary is enabled.
Title check ✅ Passed The title 'chore: add operator make commands' accurately summarizes the primary change—adding new Makefile targets and help commands for local Kubernetes operator development with kind.
Docstring Coverage ✅ Passed No functions found in the changed files to evaluate docstring coverage. Skipping docstring coverage check.
Linked Issues check ✅ Passed Check skipped because no linked issues were found for this pull request.
Out of Scope Changes check ✅ Passed Check skipped because no linked issues were found for this pull request.

✏️ Tip: You can configure your own custom pre-merge checks in the settings.

✨ Finishing Touches
🧪 Generate unit tests (beta)
  • Create PR with unit tests
  • Commit unit tests in branch feat-add-operator-make-commands

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

Comment @coderabbitai help to get the list of available commands and usage tips.

@coderabbitai coderabbitai Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Actionable comments posted: 1

Caution

Some comments are outside the diff and can’t be posted inline due to platform limitations.

⚠️ Outside diff range comments (1)
Makefile (1)

97-104: ⚠️ Potential issue | 🟡 Minor | ⚡ Quick win

Declare new kind targets as phony.

kind-load-app-images and kind-build-load-apps are command targets but are not listed in .PHONY (Line 97 onward). If matching files appear, make can skip execution unexpectedly.

Suggested fix
 .PHONY: help check_tools help_docker help_dev help_test help_local help_utils help_operator \
+       kind-load-app-images kind-build-load-apps \
        dev dev-cpu dev-local dev-local-cpu dev-local-build-lf dev-local-build-lf-cpu stop clean build logs \

Also applies to: 702-712

🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

In `@Makefile` around lines 97 - 104, The Makefile defines command targets
kind-load-app-images and kind-build-load-apps but they are missing from the
.PHONY lists, so add these exact target names to the top .PHONY grouping (and
also to the later .PHONY block around the other kind/dev targets at the second
section) to ensure make always executes them even if files with those names
exist; update the .PHONY entries that currently include dev, build, logs, etc.
to include kind-load-app-images and kind-build-load-apps.
🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

Inline comments:
In `@Makefile`:
- Around line 202-203: Help text in the Makefile prints the hardcoded cluster
name "openrag" in multiple places; replace those literals with the variable
reference $(KIND_CLUSTER_NAME) so examples reflect the actual configurable
value. Update the echo lines that show the cluster name (the line that currently
prints "KIND_CLUSTER_NAME=$(KIND_CLUSTER_NAME) (default: openrag)" and the other
help/example echoes in the same help block and the sections around the ranges
mentioned) to use $(KIND_CLUSTER_NAME) in place of "openrag", ensuring all
copy/paste examples consistently reference the variable.

---

Outside diff comments:
In `@Makefile`:
- Around line 97-104: The Makefile defines command targets kind-load-app-images
and kind-build-load-apps but they are missing from the .PHONY lists, so add
these exact target names to the top .PHONY grouping (and also to the later
.PHONY block around the other kind/dev targets at the second section) to ensure
make always executes them even if files with those names exist; update the
.PHONY entries that currently include dev, build, logs, etc. to include
kind-load-app-images and kind-build-load-apps.
🪄 Autofix (Beta)

Fix all unresolved CodeRabbit comments on this PR:

  • Push a commit to this branch (recommended)
  • Create a new PR with the fixes

ℹ️ Review info
⚙️ Run configuration

Configuration used: Path: .coderabbit.yaml

Review profile: CHILL

Plan: Pro

Run ID: 99b6bb23-0160-4280-8695-4041730f5b4e

📥 Commits

Reviewing files that changed from the base of the PR and between d255040 and f9d514a.

📒 Files selected for processing (4)
  • Makefile
  • kubernetes/operator/README.md
  • kubernetes/operator/config/samples/openrag_v1alpha1_openrag-kind-local.yaml
  • kubernetes/operator/config/samples/openrag_v1alpha1_openrag.yaml

Comment thread Makefile
Comment on lines +202 to +203
@echo " $(CYAN)KIND_CLUSTER_NAME$(NC)=$(KIND_CLUSTER_NAME) (default: openrag)"
@echo ''

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

⚠️ Potential issue | 🟡 Minor | ⚡ Quick win

Use $(KIND_CLUSTER_NAME) consistently in help output examples.

Several help lines still hardcode openrag, which conflicts with the new configurable cluster variable and can cause copy/paste mistakes when users override it.

Suggested fix
-	`@echo` "                         $(CYAN)KIND_CLUSTER_NAME$(NC)=$(KIND_CLUSTER_NAME) (default: openrag)"
+	`@echo` "                         $(CYAN)KIND_CLUSTER_NAME$(NC)=$(KIND_CLUSTER_NAME) (default: openrag)"
...
-	`@echo` "  $(PURPLE)kind load docker-image openrag-operator:dev --name openrag$(NC)"
+	`@echo` "  $(PURPLE)kind load docker-image openrag-operator:dev --name $(KIND_CLUSTER_NAME)$(NC)"
...
-	`@echo` "  1. $(CYAN)kind create cluster --name openrag$(NC)"
+	`@echo` "  1. $(CYAN)kind create cluster --name $(KIND_CLUSTER_NAME)$(NC)"
...
-	`@echo` "  $(PURPLE)make kind-build-load-apps$(NC) - Build app images and load into kind (KIND_CLUSTER_NAME=openrag)"
+	`@echo` "  $(PURPLE)make kind-build-load-apps$(NC) - Build app images and load into kind (KIND_CLUSTER_NAME=$(KIND_CLUSTER_NAME))"

Also applies to: 218-231, 287-288

🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

In `@Makefile` around lines 202 - 203, Help text in the Makefile prints the
hardcoded cluster name "openrag" in multiple places; replace those literals with
the variable reference $(KIND_CLUSTER_NAME) so examples reflect the actual
configurable value. Update the echo lines that show the cluster name (the line
that currently prints "KIND_CLUSTER_NAME=$(KIND_CLUSTER_NAME) (default:
openrag)" and the other help/example echoes in the same help block and the
sections around the ranges mentioned) to use $(KIND_CLUSTER_NAME) in place of
"openrag", ensuring all copy/paste examples consistently reference the variable.

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

This PR improves the local Kubernetes developer experience by adding root-level make targets and operator docs for building/loading OpenRAG images into a local kind cluster, plus a low-resource sample CR tailored for typical kind nodes.

Changes:

  • Added make help_operator, kind-load-app-images, and kind-build-load-apps targets to streamline kind-based workflows from the repo root.
  • Updated operator README with a documented kind + local-images workflow (including rebuild/reload + rollout restart guidance).
  • Introduced a new openrag-kind-local sample CR with reduced resource requests and imagePullPolicy: Never, and pointed the default sample at it for kind users.

Reviewed changes

Copilot reviewed 4 out of 4 changed files in this pull request and generated 2 comments.

File Description
Makefile Adds operator/kind help section and new kind image build/load targets for local clusters.
kubernetes/operator/README.md Documents the end-to-end kind workflow for running the operator and using locally loaded images.
kubernetes/operator/config/samples/openrag_v1alpha1_openrag.yaml Adds guidance to use the new kind-local sample on small clusters.
kubernetes/operator/config/samples/openrag_v1alpha1_openrag-kind-local.yaml New low-resource sample CR for kind/Colima with imagePullPolicy: Never.
Comments suppressed due to low confidence (3)

Makefile:104

  • New targets kind-load-app-images and kind-build-load-apps are not included in .PHONY. If a file with either name exists, make may skip the recipe, leading to confusing behavior. Add these targets to the .PHONY list for consistency with the rest of the Makefile.
.PHONY: help check_tools help_docker help_dev help_test help_local help_utils help_operator \
       dev dev-cpu dev-local dev-local-cpu dev-local-build-lf dev-local-build-lf-cpu stop clean build logs \
       shell-backend shell-frontend install \
       test test-unit test-integration test-ci test-ci-local test-sdk test-os-jwt lint \
       backend frontend docling docling-stop install-be install-fe build-be build-fe build-os build-lf logs-be logs-fe logs-lf logs-os \
       shell-be shell-lf shell-os restart status health db-reset clear-os-data flow-upload setup factory-reset \
       dev-branch build-langflow-dev stop-dev clean-dev logs-dev logs-lf-dev shell-lf-dev restart-dev status-dev \
       ensure-langflow-data ensure-backend-volumes

Makefile:288

  • In help_docker, the description for kind-build-load-apps hardcodes KIND_CLUSTER_NAME=openrag. Since KIND_CLUSTER_NAME is configurable, the help line should reference $(KIND_CLUSTER_NAME) and/or mention the default value to avoid misleading output.
	@echo "  $(PURPLE)make build-be$(NC)        - Build backend Docker image only"
	@echo "  $(PURPLE)make build-fe$(NC)        - Build frontend Docker image only"
	@echo "  $(PURPLE)make build-lf$(NC)        - Build Langflow Docker image only"
	@echo "  $(PURPLE)make kind-build-load-apps$(NC) - Build app images and load into kind (KIND_CLUSTER_NAME=openrag)"
	@echo "  $(PURPLE)make kind-load-app-images$(NC) - Load already-built app images into kind"

kubernetes/operator/README.md:164

  • The new kind workflow section hardcodes the cluster name openrag (e.g., kind load docker-image ... --name openrag and kind delete cluster --name openrag). Since users may create clusters with different names, consider documenting a variable/placeholder (or referencing KIND_CLUSTER_NAME) to keep the instructions correct for non-default setups.
Optional: build/load the **operator** image the same way:

```bash
cd kubernetes/operator
make docker-build IMG=openrag-operator:dev
kind load docker-image openrag-operator:dev --name openrag
make deploy IMG=openrag-operator:dev   # instead of make run

5. Tear down

kind delete cluster --name openrag
</details>



---

💡 <a href="/langflow-ai/openrag/new/main?filename=.github/instructions/*.instructions.md" class="Link--inTextBlock" target="_blank" rel="noopener noreferrer">Add Copilot custom instructions</a> for smarter, more guided reviews. <a href="https://docs.github.com/en/copilot/customizing-copilot/adding-repository-custom-instructions-for-github-copilot" class="Link--inTextBlock" target="_blank" rel="noopener noreferrer">Learn how to get started</a>.

Comment thread Makefile
Comment on lines +214 to +231
@echo "$(PURPLE)Operator in-cluster:$(NC)"
@echo " $(PURPLE)cd $(OPERATOR_DIR) && make deploy$(NC) - Deploy operator (IMG=...)"
@echo " $(PURPLE)cd $(OPERATOR_DIR) && make undeploy$(NC) - Remove operator deployment"
@echo " $(PURPLE)cd $(OPERATOR_DIR) && make docker-build$(NC) - Build operator image (IMG=...)"
@echo " $(PURPLE)kind load docker-image openrag-operator:dev --name openrag$(NC)"
@echo " $(PURPLE)helm install openrag-operator ./kubernetes/helm/operator -n openrag-control --create-namespace$(NC)"
@echo ''
@echo "$(PURPLE)Sample OpenRAG CR (after make run or make deploy):$(NC)"
@echo " $(PURPLE)kubectl create namespace my-tenant$(NC)"
@echo " $(PURPLE)kubectl apply -f $(OPERATOR_DIR)/config/samples/openrag_v1alpha1_openrag-kind-local.yaml$(NC)"
@echo " (low CPU + imagePullPolicy: Never for local images)"
@echo " $(PURPLE)kubectl get pods -n my-tenant$(NC)"
@echo " $(PURPLE)kubectl rollout restart deployment -n my-tenant openrag-fe openrag-be openrag-lf$(NC)"
@echo " (after rebuilding and reloading images)"
@echo ''
@echo "$(PURPLE)Typical kind + local images workflow:$(NC)"
@echo " 1. $(CYAN)kind create cluster --name openrag$(NC)"
@echo " 2. $(CYAN)make kind-build-load-apps$(NC)"
Comment thread Makefile
Comment on lines +699 to +707
# kind cluster name for local Kubernetes (see kubernetes/operator/README.md)
KIND_CLUSTER_NAME ?= openrag

kind-load-app-images: ## Load OpenRAG app images into a kind cluster (Colima/Docker)
@command -v kind >/dev/null 2>&1 || { echo "$(RED)kind is not installed$(NC)"; exit 1; }
@echo "$(YELLOW)Loading app images into kind cluster '$(KIND_CLUSTER_NAME)'...$(NC)"
kind load docker-image langflowai/openrag-backend:latest --name $(KIND_CLUSTER_NAME)
kind load docker-image langflowai/openrag-frontend:latest --name $(KIND_CLUSTER_NAME)
kind load docker-image langflowai/openrag-langflow:latest --name $(KIND_CLUSTER_NAME)
@github-actions github-actions Bot added the lgtm label May 19, 2026
@edwinjosechittilappilly edwinjosechittilappilly merged commit 89e906d into main May 19, 2026
22 of 23 checks passed
@github-actions github-actions Bot deleted the feat-add-operator-make-commands branch May 19, 2026 21:11
ricofurtado pushed a commit that referenced this pull request May 20, 2026
* Add kind build/load targets and kind sample

Add Makefile targets to build and load app images into a kind cluster (KIND_CLUSTER_NAME default: openrag): kind-load-app-images and kind-build-load-apps, and expose them in the help output. Update kubernetes/operator/README.md with instructions to run the operator locally, apply a kind-local sample, build/load images into kind, and restart deployments after rebuilding. Add a new config sample openrag_v1alpha1_openrag-kind-local.yaml tuned for 2-CPU kind/Colima clusters (lower resource requests, imagePullPolicy: Never). Also annotate the default sample to point users to the kind-local file for local clusters.

* Add Makefile help for operator & kind

Add a new help_operator target and OPERATOR_DIR variable to the Makefile, and include it in the .PHONY list. The new help output documents Kubernetes operator and kind-related commands (build/load app images into kind, operator deps/install/run/build/test/lint/manifests/generate, deploy/undeploy, docker-build, helm install) and provides a typical kind + local images workflow and sample CR usage. This makes it easier for developers to discover and run local operator/cluster tasks from the repo root.
ricofurtado pushed a commit that referenced this pull request May 22, 2026
* Add kind build/load targets and kind sample

Add Makefile targets to build and load app images into a kind cluster (KIND_CLUSTER_NAME default: openrag): kind-load-app-images and kind-build-load-apps, and expose them in the help output. Update kubernetes/operator/README.md with instructions to run the operator locally, apply a kind-local sample, build/load images into kind, and restart deployments after rebuilding. Add a new config sample openrag_v1alpha1_openrag-kind-local.yaml tuned for 2-CPU kind/Colima clusters (lower resource requests, imagePullPolicy: Never). Also annotate the default sample to point users to the kind-local file for local clusters.

* Add Makefile help for operator & kind

Add a new help_operator target and OPERATOR_DIR variable to the Makefile, and include it in the .PHONY list. The new help output documents Kubernetes operator and kind-related commands (build/load app images into kind, operator deps/install/run/build/test/lint/manifests/generate, deploy/undeploy, docker-build, helm install) and provides a typical kind + local images workflow and sample CR usage. This makes it easier for developers to discover and run local operator/cluster tasks from the repo root.
ricofurtado added a commit that referenced this pull request May 22, 2026
…on (#1625)

* feat: Implement in-process JWT claims caching with TTL and LRU eviction

* style: ruff format (auto)

* refactor: Update cache initialization to use settings for max size and TTL

* feat: code cleanup

* fix: Improve public key file reading and enhance token scheme handling

* style: ruff autofix (auto)

* fix: Handle user insert races and add test timeout (#1618)

* Handle user insert races and add test timeout

Increase TypeScript integration test timeout to 120s to reduce flakiness during slow CI runs.

Enhance user_service.ensure_user_row IntegrityError handling to explicitly handle concurrent-insert races: detect a (oauth_provider, oauth_subject) race and return the existing row, detect an email_lookup_hash race by looking up the email and returning the concurrent identity when it matches, and handle PK collisions by retrying the insert with a new UUID. Add explanatory comments about which collisions are recoverable and when errors should propagate to the caller.

* style: ruff format (auto)

* Use PEP 604 union for agent config return type

Replace typing.Optional[dict] with PEP 604 union syntax (dict | None) for get_effective_agent_config and remove the now-unused Optional import. This is a pure type-annotation cleanup (no runtime behavior changes); note it requires Python 3.10+ for the `|` union syntax.

---------

Co-authored-by: autofix-ci[bot] <114827586+autofix-ci[bot]@users.noreply.github.com>

* fix: display file names and fix ingestion for onedrive (#1609)

* fixed onedrive not working

* style: ruff format (auto)

* removed sites.read.all from onedrive

* style: ruff format (auto)

* fixed allowed users and groups

* fix lint error

* fixed lint

* fixed mypy lint

* style: ruff format (auto)

---------

Co-authored-by: autofix-ci[bot] <114827586+autofix-ci[bot]@users.noreply.github.com>

* fix: Skip deleted connector files; return orphan IDs (#1592)

* Skip deleted connector files; return orphan IDs

Handle files deleted at source and improve orphan reconciliation.

- Add TaskStatus.SKIPPED and update ConnectorFileProcessor to catch FileNotFoundError/ValueError from connector.get_file_content. When a file is missing, mark the file task as SKIPPED, set a skipped result, update timestamps and counters, and avoid raising so the upload can continue.
- Change reconcile_orphans_for_connector_type to return a list of orphan file IDs ([]) instead of an int, update early returns and docstring to reflect returning the deleted/removed document IDs so callers can exclude them from subsequent sync passes.
- Minor whitespace cleanup in sync_all_connectors.

These changes avoid aborting syncs on missing source files and make orphan deletions explicit for downstream processing.

* Skip deleted connector files; return orphan IDs

Handle files deleted at source and improve orphan reconciliation.

- Add TaskStatus.SKIPPED and update ConnectorFileProcessor to catch FileNotFoundError/ValueError from connector.get_file_content. When a file is missing, mark the file task as SKIPPED, set a skipped result, update timestamps and counters, and avoid raising so the upload can continue.
- Change reconcile_orphans_for_connector_type to return a list of orphan file IDs ([]) instead of an int, update early returns and docstring to reflect returning the deleted/removed document IDs so callers can exclude them from subsequent sync passes.
- Minor whitespace cleanup in sync_all_connectors.

These changes avoid aborting syncs on missing source files and make orphan deletions explicit for downstream processing.

* Add *.db to .gitignore

Add a '*.db' pattern to .gitignore to prevent local database files (e.g., SQLite) from being committed to the repository.

* Add orphan reconcile and bulk-delete helper

Introduce a reconcile pass and supporting bulk-delete utility to remove OpenSearch chunks for files that no longer exist remotely.

- Add reconcile_orphans_for_connector_type (src/api/connectors.py): lists all active connections for a connector type, strictly gates on unauthenticated connections or listing errors (abort with 0 deletes), aggregates paginated remote file IDs, computes orphans (indexed IDs not present remotely) and invokes delete_chunks_by_document_ids. Integrated into connector_sync and sync_all_connectors (skips reconcile when sync is capped).
- Add delete_chunks_by_document_ids (src/api/documents.py): issues a single delete_by_query with terms(document_id, ...) and conflicts="proceed", returns deleted count, and short-circuits on empty input.
- Update connector metadata painless params (src/connectors/service.py): include filename param and assign ctx._source.filename = params.filename in the update script so renamed files update indexed chunks.
- Add unit tests: cover delete_chunks_by_document_ids, reconcile_orphans_for_connector_type (gating, pagination, multi-connection union, error handling), and connector metadata filename behavior.

Behavior notes: reconcile is conservative to avoid false-positive deletions; bulk delete is defensive and returns 0 on unexpected responses or failures.

* fix lint

* style: ruff format (auto)

* fix lint

* Use max_files param and add typing

Add an explicit type annotation for files_to_process and update the connector.list_files call to use the max_files parameter instead of limit to match the connector API. Also remove redundant `# type: ignore` comments on connector.cfg assignments as they are no longer necessary. Minor cleanup to improve type clarity and API consistency.

* Add connector sync preview UI and API

Introduce sync-preview functionality for connectors and sync-all flows. Backend: add ID→filename aggregation, compute_orphans (non-destructive orphan detection), orphan deletion helper, and preview endpoints (connector_sync_preview, connectors_sync_all_preview) plus wiring in internal routes. Frontend: add useSyncConnectorPreview/useSyncAllConnectorsPreview mutations, wire preview flows into Knowledge pages/components, and add SyncConfirmDialog component to show orphan lists, per-connector synced counts, availability flags, and confirm deletion+sync interactions. Also update UI buttons to open the preview dialog and handle loading/syncing states.

* style: ruff format (auto)

* Hoist and consolidate imports in processors

Move many inline imports to module scope and consolidate repeated imports across TaskProcessor and its subclasses. Adds top-level imports (asyncio, datetime, mimetypes, os, time), config/settings and utility functions (clients, get_embedding_model, get_index_name, get_openrag_config, extract_relevant, process_text_file, resplit_chunks_character_windows, ensure_embedding_field_exists, auto_cleanup_tempfile, hash_id, build_filename_delete_body, build_filename_search_body) and imports TaskStatus. Removes redundant per-function imports to improve readability and reduce repeated import overhead; no functional changes intended.

* Type UseQueryOptions with SearchResult

Specify UseQueryOptions<SearchResult> for the useGetSearchQuery hook's options parameter to improve TypeScript type inference and ensure the query options expect SearchResult data. This change tightens typings without altering runtime behavior.

---------

Co-authored-by: autofix-ci[bot] <114827586+autofix-ci[bot]@users.noreply.github.com>

* fix: add reset db on command, expose error on e2e tests (#1627)

* fix factory reset not deleting data

* make onboarding errors pop up on e2e tests

* style: ruff format (auto)

* added check for error when uploading

* added check if its on first step to rollback, not only if its complete

* reset correctly

* update timeout for uploading document

* remove misclick

* fixed lint

* fix mypy errors

* style: ruff format (auto)

---------

Co-authored-by: autofix-ci[bot] <114827586+autofix-ci[bot]@users.noreply.github.com>

* refactor: handler for file upload for context (#1624)

* file upload for context

* coderabbit suggestions

* chore:  add operator make commands (#1628)

* Add kind build/load targets and kind sample

Add Makefile targets to build and load app images into a kind cluster (KIND_CLUSTER_NAME default: openrag): kind-load-app-images and kind-build-load-apps, and expose them in the help output. Update kubernetes/operator/README.md with instructions to run the operator locally, apply a kind-local sample, build/load images into kind, and restart deployments after rebuilding. Add a new config sample openrag_v1alpha1_openrag-kind-local.yaml tuned for 2-CPU kind/Colima clusters (lower resource requests, imagePullPolicy: Never). Also annotate the default sample to point users to the kind-local file for local clusters.

* Add Makefile help for operator & kind

Add a new help_operator target and OPERATOR_DIR variable to the Makefile, and include it in the .PHONY list. The new help output documents Kubernetes operator and kind-related commands (build/load app images into kind, operator deps/install/run/build/test/lint/manifests/generate, deploy/undeploy, docker-build, helm install) and provides a typical kind + local images workflow and sample CR usage. This makes it easier for developers to discover and run local operator/cluster tasks from the repo root.

* redirect on logout (#1440) (#1636)

* fix: Copy flows directory into Docker image (#1632)

Add a COPY instruction to include the local flows/ directory at /app/flows in the image so bundled flows are available at runtime. Placed before the entrypoint/ownership handling to ensure the files are copied with the intended UID/permissions.

* feat: settings saas tabs menu (#1637)

* remove logic for previous tab selected, need to be discussed with Ana

* reduce repeated code and use darkmodelight class

* reduce repeated code and use darkmodelight class

* remove unecessary !important

* remove unecessary !important

* remove duplicate css blocks

* remove !important and merge active and hover state

* put back TabsContent anf fix hover issue

---------

Co-authored-by: Olfa Maslah <[email protected]>

* chore: Add ruff autofix step to CI workflows (#1640)

* Add ruff autofix step to CI workflows

Add an in-place Ruff fix step to .github/workflows/autofix.ci.yml that runs uv run ruff check --fix on changed files, rename the job and autofix commit message to reflect "ruff autofix", and update comments to clarify that autofix.ci applies safe fixes and formatting. Also update .github/workflows/lint-backend.yml comments to state that safe lint fixes and formatting are auto-applied by autofix.ci while keeping strict lint (--no-fix) and mypy in the lint workflow.

* Update autofix.ci.yml

* fix: Enum IDs and delete OpenSearch docs by _id (#1638)

* Enum IDs and delete OpenSearch docs by _id

Add DLS-safe OpenSearch delete helpers and use them to avoid silent no-ops from delete_by_query. Introduces utils/opensearch_delete.py with collect_visible_document_ids and delete_document_ids (enumerate visible _ids via search/scroll, then delete by primary _id). Update delete_chunks_by_document_ids to enumerate chunk IDs and delete each by _id. Ensure langflow_connector_service and TaskProcessor clear stale chunks (by document_id) before re-ingest/re-index to prevent duplicate or trailing chunks after renames. Improve connector listing by scoping cfg.file_ids/folder_ids when available to avoid false orphan detection. Add and update unit tests to assert the new enumeration-and-delete behavior and ordering.

* style: ruff format (auto)

* ruff fix

* ruff fix

---------

Co-authored-by: autofix-ci[bot] <114827586+autofix-ci[bot]@users.noreply.github.com>

* make probes more robust (#1642)

* feat: Show google_drive connector for cloud brand (#1650)

Remove google_drive from the cloud-brand exclusion so Google Drive connectors are visible when isCloudBrand is true. Applied the change in connector-cards.tsx and knowledge-dropdown.tsx (OneDrive remains excluded).

* operator ubi base build (#1652)

* remove flow download initContainer when flowRef is removed (#1653)

* fix: cr deletion (#1656)

* fix cr deletion

* refactoring

* fix unit test

* fix lint

* fix cross namespace CR deletion

* Potential fix for pull request finding 'CodeQL / Information exposure through an exception'

Co-authored-by: Copilot Autofix powered by AI <62310815+github-advanced-security[bot]@users.noreply.github.com>

---------

Co-authored-by: autofix-ci[bot] <114827586+autofix-ci[bot]@users.noreply.github.com>
Co-authored-by: Edwin Jose <[email protected]>
Co-authored-by: Lucas Oliveira <[email protected]>
Co-authored-by: Mike Fortman <[email protected]>
Co-authored-by: Wallgau <[email protected]>
Co-authored-by: Olfa Maslah <[email protected]>
Co-authored-by: ming <[email protected]>
Co-authored-by: Copilot Autofix powered by AI <62310815+github-advanced-security[bot]@users.noreply.github.com>
ricofurtado pushed a commit that referenced this pull request May 23, 2026
* Add kind build/load targets and kind sample

Add Makefile targets to build and load app images into a kind cluster (KIND_CLUSTER_NAME default: openrag): kind-load-app-images and kind-build-load-apps, and expose them in the help output. Update kubernetes/operator/README.md with instructions to run the operator locally, apply a kind-local sample, build/load images into kind, and restart deployments after rebuilding. Add a new config sample openrag_v1alpha1_openrag-kind-local.yaml tuned for 2-CPU kind/Colima clusters (lower resource requests, imagePullPolicy: Never). Also annotate the default sample to point users to the kind-local file for local clusters.

* Add Makefile help for operator & kind

Add a new help_operator target and OPERATOR_DIR variable to the Makefile, and include it in the .PHONY list. The new help output documents Kubernetes operator and kind-related commands (build/load app images into kind, operator deps/install/run/build/test/lint/manifests/generate, deploy/undeploy, docker-build, helm install) and provides a typical kind + local images workflow and sample CR usage. This makes it easier for developers to discover and run local operator/cluster tasks from the repo root.
ricofurtado added a commit that referenced this pull request May 23, 2026
…on (#1625)

* feat: Implement in-process JWT claims caching with TTL and LRU eviction

* style: ruff format (auto)

* refactor: Update cache initialization to use settings for max size and TTL

* feat: code cleanup

* fix: Improve public key file reading and enhance token scheme handling

* style: ruff autofix (auto)

* fix: Handle user insert races and add test timeout (#1618)

* Handle user insert races and add test timeout

Increase TypeScript integration test timeout to 120s to reduce flakiness during slow CI runs.

Enhance user_service.ensure_user_row IntegrityError handling to explicitly handle concurrent-insert races: detect a (oauth_provider, oauth_subject) race and return the existing row, detect an email_lookup_hash race by looking up the email and returning the concurrent identity when it matches, and handle PK collisions by retrying the insert with a new UUID. Add explanatory comments about which collisions are recoverable and when errors should propagate to the caller.

* style: ruff format (auto)

* Use PEP 604 union for agent config return type

Replace typing.Optional[dict] with PEP 604 union syntax (dict | None) for get_effective_agent_config and remove the now-unused Optional import. This is a pure type-annotation cleanup (no runtime behavior changes); note it requires Python 3.10+ for the `|` union syntax.

---------

Co-authored-by: autofix-ci[bot] <114827586+autofix-ci[bot]@users.noreply.github.com>

* fix: display file names and fix ingestion for onedrive (#1609)

* fixed onedrive not working

* style: ruff format (auto)

* removed sites.read.all from onedrive

* style: ruff format (auto)

* fixed allowed users and groups

* fix lint error

* fixed lint

* fixed mypy lint

* style: ruff format (auto)

---------

Co-authored-by: autofix-ci[bot] <114827586+autofix-ci[bot]@users.noreply.github.com>

* fix: Skip deleted connector files; return orphan IDs (#1592)

* Skip deleted connector files; return orphan IDs

Handle files deleted at source and improve orphan reconciliation.

- Add TaskStatus.SKIPPED and update ConnectorFileProcessor to catch FileNotFoundError/ValueError from connector.get_file_content. When a file is missing, mark the file task as SKIPPED, set a skipped result, update timestamps and counters, and avoid raising so the upload can continue.
- Change reconcile_orphans_for_connector_type to return a list of orphan file IDs ([]) instead of an int, update early returns and docstring to reflect returning the deleted/removed document IDs so callers can exclude them from subsequent sync passes.
- Minor whitespace cleanup in sync_all_connectors.

These changes avoid aborting syncs on missing source files and make orphan deletions explicit for downstream processing.

* Skip deleted connector files; return orphan IDs

Handle files deleted at source and improve orphan reconciliation.

- Add TaskStatus.SKIPPED and update ConnectorFileProcessor to catch FileNotFoundError/ValueError from connector.get_file_content. When a file is missing, mark the file task as SKIPPED, set a skipped result, update timestamps and counters, and avoid raising so the upload can continue.
- Change reconcile_orphans_for_connector_type to return a list of orphan file IDs ([]) instead of an int, update early returns and docstring to reflect returning the deleted/removed document IDs so callers can exclude them from subsequent sync passes.
- Minor whitespace cleanup in sync_all_connectors.

These changes avoid aborting syncs on missing source files and make orphan deletions explicit for downstream processing.

* Add *.db to .gitignore

Add a '*.db' pattern to .gitignore to prevent local database files (e.g., SQLite) from being committed to the repository.

* Add orphan reconcile and bulk-delete helper

Introduce a reconcile pass and supporting bulk-delete utility to remove OpenSearch chunks for files that no longer exist remotely.

- Add reconcile_orphans_for_connector_type (src/api/connectors.py): lists all active connections for a connector type, strictly gates on unauthenticated connections or listing errors (abort with 0 deletes), aggregates paginated remote file IDs, computes orphans (indexed IDs not present remotely) and invokes delete_chunks_by_document_ids. Integrated into connector_sync and sync_all_connectors (skips reconcile when sync is capped).
- Add delete_chunks_by_document_ids (src/api/documents.py): issues a single delete_by_query with terms(document_id, ...) and conflicts="proceed", returns deleted count, and short-circuits on empty input.
- Update connector metadata painless params (src/connectors/service.py): include filename param and assign ctx._source.filename = params.filename in the update script so renamed files update indexed chunks.
- Add unit tests: cover delete_chunks_by_document_ids, reconcile_orphans_for_connector_type (gating, pagination, multi-connection union, error handling), and connector metadata filename behavior.

Behavior notes: reconcile is conservative to avoid false-positive deletions; bulk delete is defensive and returns 0 on unexpected responses or failures.

* fix lint

* style: ruff format (auto)

* fix lint

* Use max_files param and add typing

Add an explicit type annotation for files_to_process and update the connector.list_files call to use the max_files parameter instead of limit to match the connector API. Also remove redundant `# type: ignore` comments on connector.cfg assignments as they are no longer necessary. Minor cleanup to improve type clarity and API consistency.

* Add connector sync preview UI and API

Introduce sync-preview functionality for connectors and sync-all flows. Backend: add ID→filename aggregation, compute_orphans (non-destructive orphan detection), orphan deletion helper, and preview endpoints (connector_sync_preview, connectors_sync_all_preview) plus wiring in internal routes. Frontend: add useSyncConnectorPreview/useSyncAllConnectorsPreview mutations, wire preview flows into Knowledge pages/components, and add SyncConfirmDialog component to show orphan lists, per-connector synced counts, availability flags, and confirm deletion+sync interactions. Also update UI buttons to open the preview dialog and handle loading/syncing states.

* style: ruff format (auto)

* Hoist and consolidate imports in processors

Move many inline imports to module scope and consolidate repeated imports across TaskProcessor and its subclasses. Adds top-level imports (asyncio, datetime, mimetypes, os, time), config/settings and utility functions (clients, get_embedding_model, get_index_name, get_openrag_config, extract_relevant, process_text_file, resplit_chunks_character_windows, ensure_embedding_field_exists, auto_cleanup_tempfile, hash_id, build_filename_delete_body, build_filename_search_body) and imports TaskStatus. Removes redundant per-function imports to improve readability and reduce repeated import overhead; no functional changes intended.

* Type UseQueryOptions with SearchResult

Specify UseQueryOptions<SearchResult> for the useGetSearchQuery hook's options parameter to improve TypeScript type inference and ensure the query options expect SearchResult data. This change tightens typings without altering runtime behavior.

---------

Co-authored-by: autofix-ci[bot] <114827586+autofix-ci[bot]@users.noreply.github.com>

* fix: add reset db on command, expose error on e2e tests (#1627)

* fix factory reset not deleting data

* make onboarding errors pop up on e2e tests

* style: ruff format (auto)

* added check for error when uploading

* added check if its on first step to rollback, not only if its complete

* reset correctly

* update timeout for uploading document

* remove misclick

* fixed lint

* fix mypy errors

* style: ruff format (auto)

---------

Co-authored-by: autofix-ci[bot] <114827586+autofix-ci[bot]@users.noreply.github.com>

* refactor: handler for file upload for context (#1624)

* file upload for context

* coderabbit suggestions

* chore:  add operator make commands (#1628)

* Add kind build/load targets and kind sample

Add Makefile targets to build and load app images into a kind cluster (KIND_CLUSTER_NAME default: openrag): kind-load-app-images and kind-build-load-apps, and expose them in the help output. Update kubernetes/operator/README.md with instructions to run the operator locally, apply a kind-local sample, build/load images into kind, and restart deployments after rebuilding. Add a new config sample openrag_v1alpha1_openrag-kind-local.yaml tuned for 2-CPU kind/Colima clusters (lower resource requests, imagePullPolicy: Never). Also annotate the default sample to point users to the kind-local file for local clusters.

* Add Makefile help for operator & kind

Add a new help_operator target and OPERATOR_DIR variable to the Makefile, and include it in the .PHONY list. The new help output documents Kubernetes operator and kind-related commands (build/load app images into kind, operator deps/install/run/build/test/lint/manifests/generate, deploy/undeploy, docker-build, helm install) and provides a typical kind + local images workflow and sample CR usage. This makes it easier for developers to discover and run local operator/cluster tasks from the repo root.

* redirect on logout (#1440) (#1636)

* fix: Copy flows directory into Docker image (#1632)

Add a COPY instruction to include the local flows/ directory at /app/flows in the image so bundled flows are available at runtime. Placed before the entrypoint/ownership handling to ensure the files are copied with the intended UID/permissions.

* feat: settings saas tabs menu (#1637)

* remove logic for previous tab selected, need to be discussed with Ana

* reduce repeated code and use darkmodelight class

* reduce repeated code and use darkmodelight class

* remove unecessary !important

* remove unecessary !important

* remove duplicate css blocks

* remove !important and merge active and hover state

* put back TabsContent anf fix hover issue

---------

Co-authored-by: Olfa Maslah <[email protected]>

* chore: Add ruff autofix step to CI workflows (#1640)

* Add ruff autofix step to CI workflows

Add an in-place Ruff fix step to .github/workflows/autofix.ci.yml that runs uv run ruff check --fix on changed files, rename the job and autofix commit message to reflect "ruff autofix", and update comments to clarify that autofix.ci applies safe fixes and formatting. Also update .github/workflows/lint-backend.yml comments to state that safe lint fixes and formatting are auto-applied by autofix.ci while keeping strict lint (--no-fix) and mypy in the lint workflow.

* Update autofix.ci.yml

* fix: Enum IDs and delete OpenSearch docs by _id (#1638)

* Enum IDs and delete OpenSearch docs by _id

Add DLS-safe OpenSearch delete helpers and use them to avoid silent no-ops from delete_by_query. Introduces utils/opensearch_delete.py with collect_visible_document_ids and delete_document_ids (enumerate visible _ids via search/scroll, then delete by primary _id). Update delete_chunks_by_document_ids to enumerate chunk IDs and delete each by _id. Ensure langflow_connector_service and TaskProcessor clear stale chunks (by document_id) before re-ingest/re-index to prevent duplicate or trailing chunks after renames. Improve connector listing by scoping cfg.file_ids/folder_ids when available to avoid false orphan detection. Add and update unit tests to assert the new enumeration-and-delete behavior and ordering.

* style: ruff format (auto)

* ruff fix

* ruff fix

---------

Co-authored-by: autofix-ci[bot] <114827586+autofix-ci[bot]@users.noreply.github.com>

* make probes more robust (#1642)

* feat: Show google_drive connector for cloud brand (#1650)

Remove google_drive from the cloud-brand exclusion so Google Drive connectors are visible when isCloudBrand is true. Applied the change in connector-cards.tsx and knowledge-dropdown.tsx (OneDrive remains excluded).

* operator ubi base build (#1652)

* remove flow download initContainer when flowRef is removed (#1653)

* fix: cr deletion (#1656)

* fix cr deletion

* refactoring

* fix unit test

* fix lint

* fix cross namespace CR deletion

* Potential fix for pull request finding 'CodeQL / Information exposure through an exception'

Co-authored-by: Copilot Autofix powered by AI <62310815+github-advanced-security[bot]@users.noreply.github.com>

---------

Co-authored-by: autofix-ci[bot] <114827586+autofix-ci[bot]@users.noreply.github.com>
Co-authored-by: Edwin Jose <[email protected]>
Co-authored-by: Lucas Oliveira <[email protected]>
Co-authored-by: Mike Fortman <[email protected]>
Co-authored-by: Wallgau <[email protected]>
Co-authored-by: Olfa Maslah <[email protected]>
Co-authored-by: ming <[email protected]>
Co-authored-by: Copilot Autofix powered by AI <62310815+github-advanced-security[bot]@users.noreply.github.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants