Skip to content

fix: centralize backend paths and accept env var to define them, change deployments to centralize read only and writable volumes#1342

Merged
lucaseduoli merged 15 commits into
release-saas-0.1from
fix/backend_data_paths
Apr 7, 2026
Merged

fix: centralize backend paths and accept env var to define them, change deployments to centralize read only and writable volumes#1342
lucaseduoli merged 15 commits into
release-saas-0.1from
fix/backend_data_paths

Conversation

@lucaseduoli

@lucaseduoli lucaseduoli commented Apr 6, 2026

Copy link
Copy Markdown
Collaborator

This pull request introduces a major storage refactoring and breaking changes to the OpenRAG Helm chart, consolidating several PersistentVolumeClaims (PVCs) and updating the configuration and deployment logic accordingly. It also adds a migration job to help users upgrade safely and updates the Helm chart version to 0.2.0. If you are upgrading from a previous version, please follow the migration instructions to avoid data loss.

Major storage refactor and migration:

  • Consolidated legacy PVCs (config, keys, documents) into new data and shared PVCs, simplifying volume management for the backend and Langflow components. Old PVCs are deprecated and should be deleted after migration. (kubernetes/helm/openrag/templates/backend/deployment.yaml [1] [2]; kubernetes/helm/openrag/templates/langflow/deployment.yaml [3] [4]; kubernetes/helm/openrag/templates/storage/backend-sc.yaml [5] [6]; kubernetes/helm/openrag/templates/storage/config-pvc.yaml [7]; kubernetes/helm/openrag/templates/storage/keys-pvc.yaml [8]; kubernetes/helm/openrag/templates/storage/data-pvc.yaml [9]
  • Added a migration job (storage-migration) to copy existing config and keys to the new data volume during upgrade. This job must be enabled in values.yaml when upgrading from older versions. (kubernetes/helm/openrag/templates/storage/migration-job.yaml [1]; kubernetes/helm/openrag/templates/NOTES.txt [2]

Deployment and configuration updates:

  • Updated backend and Langflow deployments to mount the new data and shared volumes, and removed legacy volume mounts and related init containers. (kubernetes/helm/openrag/templates/backend/deployment.yaml [1] [2]; kubernetes/helm/openrag/templates/langflow/deployment.yaml [3] [4]
  • Centralized path configuration using new environment variables and values in values.yaml, supporting custom directory layouts. (kubernetes/helm/openrag/templates/backend/backend-dotenv.yaml [1]; .env.example [2]; docker-compose.yml [3]

Default data loading and flows:

Versioning and documentation:

  • Bumped Helm chart version to 0.2.0 and documented the breaking changes and migration steps in the release notes. (kubernetes/helm/openrag/Chart.yaml [1]; kubernetes/helm/openrag/templates/NOTES.txt [2]

Please review the release notes and migration instructions carefully before upgrading.

@lucaseduoli lucaseduoli requested a review from mpawlow April 6, 2026 18:31
@lucaseduoli lucaseduoli self-assigned this Apr 6, 2026
@github-actions github-actions Bot added backend 🔷 Issues related to backend services (OpenSearch, Langflow, APIs) bug 🔴 Something isn't working. labels Apr 6, 2026
@github-actions github-actions Bot added bug 🔴 Something isn't working. and removed bug 🔴 Something isn't working. labels Apr 6, 2026
Comment thread kubernetes/helm/openrag/templates/backend/backend-dotenv.yaml
Comment thread kubernetes/helm/openrag/templates/backend/deployment.yaml Outdated
@github-actions github-actions Bot added bug 🔴 Something isn't working. and removed bug 🔴 Something isn't working. labels Apr 6, 2026
@lucaseduoli lucaseduoli linked an issue Apr 6, 2026 that may be closed by this pull request
3 tasks
Comment thread kubernetes/helm/openrag/templates/backend/deployment.yaml
Comment thread kubernetes/helm/openrag/templates/backend/deployment.yaml

@mpawlow mpawlow left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@lucaseduoli

Code Review 1

  • See PR comments: (a) to (g)

Comment thread kubernetes/helm/openrag/templates/backend/deployment.yaml
Comment thread kubernetes/helm/openrag/templates/storage/data-pvc.yaml
Comment thread kubernetes/helm/openrag/values.yaml
Comment thread src/config/paths.py
Comment thread src/config/paths.py
Comment thread kubernetes/helm/openrag/values.yaml
Comment thread kubernetes/helm/openrag/values.yaml
@github-actions github-actions Bot added docker bug 🔴 Something isn't working. and removed bug 🔴 Something isn't working. labels Apr 7, 2026
@github-actions github-actions Bot removed the bug 🔴 Something isn't working. label Apr 7, 2026
@github-actions github-actions Bot added the bug 🔴 Something isn't working. label Apr 7, 2026

@mpawlow mpawlow left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@lucaseduoli

Code Review 2

  • ✅ Approved / LGTM
  • See PR comments: (2a), (2b)

Comment thread kubernetes/helm/openrag/templates/storage/migration-job.yaml Outdated
Comment thread kubernetes/helm/openrag/templates/langflow/deployment.yaml
@github-actions github-actions Bot added bug 🔴 Something isn't working. and removed bug 🔴 Something isn't working. labels Apr 7, 2026
@github-actions github-actions Bot added bug 🔴 Something isn't working. and removed bug 🔴 Something isn't working. labels Apr 7, 2026
@lucaseduoli lucaseduoli merged commit 8dfcc9a into release-saas-0.1 Apr 7, 2026
6 checks passed
@github-actions github-actions Bot deleted the fix/backend_data_paths branch April 7, 2026 18:48
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

backend 🔷 Issues related to backend services (OpenSearch, Langflow, APIs) bug 🔴 Something isn't working. docker lgtm

Projects

None yet

Development

Successfully merging this pull request may close these issues.

[Feature]: Make OpenRag stateful data folder paths configurable

3 participants