Skip to content

fix: persist Langflow database across container restarts#1167

Merged
mpawlow merged 7 commits into
mainfrom
mp/fix/GH-1127-langflow-flow-edits-not-persisted
Mar 20, 2026
Merged

fix: persist Langflow database across container restarts#1167
mpawlow merged 7 commits into
mainfrom
mp/fix/GH-1127-langflow-flow-edits-not-persisted

Conversation

@mpawlow

@mpawlow mpawlow commented Mar 17, 2026

Copy link
Copy Markdown
Collaborator

Issue

Summary

  • Fixes Langflow flow edits not being persisted across container restarts by replacing the LANGFLOW_LOAD_FLOWS_PATH blind-upsert mechanism with a create-only ensure_flows_exist() approach that only seeds missing flows, preserving any user edits made in the Langflow UI
  • Corrects the Langflow database and config directory mount path so the SQLite database is actually stored on the host-mounted volume and survives container restarts

Root Cause Fix: Langflow Data Persistence

  • Changed the docker-compose.yml volume mount for Langflow from /root/.langflow (the old default home directory) to /app/langflow-data (the path the langflow user actually owns inside the container), ensuring the SQLite database is written to the host-mounted volume
  • Replaced the LANGFLOW_LOAD_FLOWS_PATH env var with LANGFLOW_CONFIG_DIR, which sets Langflow's config directory rather than triggering a blind flow upsert on every startup
  • Made LANGFLOW_DATABASE_URL configurable via .env with a sensible default; supports override with a PostgreSQL URL for production deployments
  • Pre-creates the /app/langflow-data directory in Dockerfile.langflow so it is owned by the correct UID (1000) before the volume is mounted

New ensure_flows_exist() Behavior

  • Added FlowsService.ensure_flows_exist() — a create-only method called at startup that checks whether each configured flow exists in the Langflow database (via GET /api/v1/flows/{id}) and only creates it (via PUT) if a 404 is returned
  • Existing flows are never patched or overwritten, preserving user edits made in the Langflow UI
  • Replaces LANGFLOW_LOAD_FLOWS_PATH, which performed a blind upsert on every container start and silently discarded user customizations
  • startup_tasks() in src/main.py now calls ensure_flows_exist() before check_flows_reset(), and filters newly-seeded flows out of the reset-detection logic to avoid false positives

Configuration & TUI

  • Added LANGFLOW_DATA_PATH as a configurable field in the TUI setup wizard (src/tui/config_fields.py, src/tui/managers/env_manager.py, src/tui/screens/config.py), with a directory picker matching the existing OpenSearch data path pattern
  • Default TUI value: $HOME/.openrag/data/langflow-data
  • LANGFLOW_DATA_PATH added to .env.example and frontend/.env.test.example
  • LANGFLOW_DATA_PATH documented in docs/docs/reference/configuration.mdx

CI / Developer Experience

  • Updated Makefile: added ensure-langflow-data target that pre-creates ./langflow-data with world-writable permissions; all dev, dev-cpu, dev-local, dev-local-cpu, dev-branch, dev-branch-cpu, test-ci, test-ci-local, and restart-dev targets depend on it
  • Updated factory-reset to also remove the langflow-data directory
  • Updated .github/workflows/test-e2e.yml and test-integration.yml cleanup steps to also delete langflow-data (root-owned files) via an Alpine container
  • Updated scripts/setup-e2e.sh to pre-create and chmod 777 the langflow-data directory before starting infrastructure
  • Added langflow-data/ to .gitignore

Kubernetes

  • Updated kubernetes/helm/openrag/templates/langflow/deployment.yaml to replace LANGFLOW_LOAD_FLOWS_PATH with LANGFLOW_CONFIG_DIR, consistent with the Docker Compose change

Builds on #1129

Co-authored by @JasonOA888

@mpawlow mpawlow self-assigned this Mar 17, 2026
@mpawlow mpawlow changed the title DRAFT: fix: persist Langflow database across container restarts fix: persist Langflow database across container restarts Mar 17, 2026
@github-actions github-actions Bot added the bug 🔴 Something isn't working. label Mar 17, 2026
@mpawlow mpawlow force-pushed the mp/fix/GH-1127-langflow-flow-edits-not-persisted branch from 2dbfb1d to 4bda6e5 Compare March 18, 2026 21:14
@github-actions github-actions Bot added backend 🔷 Issues related to backend services (OpenSearch, Langflow, APIs) bug 🔴 Something isn't working. documentation 📘 Improvements or additions to documentation frontend 🟨 Issues related to the UI/UX ci ⬛ CI/CD, build, and infrastructure issues tests and removed bug 🔴 Something isn't working. documentation 📘 Improvements or additions to documentation labels Mar 18, 2026
@github-actions

This comment has been minimized.

@github-actions github-actions Bot added bug 🔴 Something isn't working. and removed bug 🔴 Something isn't working. labels Mar 18, 2026
@mpawlow mpawlow force-pushed the mp/fix/GH-1127-langflow-flow-edits-not-persisted branch from d062003 to 82c368c Compare March 19, 2026 14:46
@github-actions github-actions Bot added documentation 📘 Improvements or additions to documentation bug 🔴 Something isn't working. and removed bug 🔴 Something isn't working. documentation 📘 Improvements or additions to documentation labels Mar 19, 2026
@github-actions

This comment has been minimized.

@mpawlow mpawlow force-pushed the mp/fix/GH-1127-langflow-flow-edits-not-persisted branch from 82c368c to 0656719 Compare March 19, 2026 15:03
@github-actions github-actions Bot added documentation 📘 Improvements or additions to documentation and removed bug 🔴 Something isn't working. labels Mar 19, 2026
JasonOA888 and others added 6 commits March 19, 2026 14:40
Fixes #1127 - Langflow flow edits no longer lost after restart

## Problem
Flow edits made in Langflow UI were being lost after container restart.
The langflow service only mounted /app/flows but Langflow stores its
SQLite database at /root/.langflow/ by default, which was ephemeral.

## Solution
1. Add persistent volume for Langflow data directory:
   - Mount ${LANGFLOW_DATA_PATH:-./langflow-data} to /root/.langflow
2. Explicitly set LANGFLOW_DATABASE_URL to ensure database location
3. Document the new LANGFLOW_DATA_PATH in .env.example

## Testing
- Verified volume mount configuration
- Database URL format: sqlite:////root/.langflow/langflow.db

Closes #1127
Issue

- #1127

Summary

- User-made edits to Langflow flows were silently discarded on every container restart.
- Two root causes were identified and corrected:
- (1) The Langflow data volume was mounted to the wrong path inside the container.
- (2) The LANGFLOW_LOAD_FLOWS_PATH mechanism performed a blind upsert of all flows on every startup, overwriting any changes made in the Langflow UI.

Docker / Infrastructure

- Corrected the Langflow data volume mount target from /root/.langflow to /app/langflow-data in docker-compose.yml
- Replaced LANGFLOW_LOAD_FLOWS_PATH env var with LANGFLOW_CONFIG_DIR=/app/langflow-data so Langflow resolves its config and database from the persisted volume
- Updated LANGFLOW_DATABASE_URL to reference the new path (sqlite:////app/langflow-data/langflow.db)
- Pre-created /app/langflow-data in Dockerfile.langflow during image build to ensure named Docker volumes are initialised with the correct ownership for the non-root container user
- Added langflow-data/ directory with a .gitkeep file; updated .gitignore to track the directory stub while ignoring its contents

Backend — Flow Bootstrapping

- Added FlowsService.ensure_flows_exist(): a create-only startup routine that checks each configured flow ID against the Langflow API and creates missing flows from their JSON files, without ever patching or
overwriting an existing flow
- Replaced the LANGFLOW_LOAD_FLOWS_PATH blind-upsert behaviour with a call to ensure_flows_exist() inside startup_tasks() in src/main.py

Makefile

- Extended the factory-reset target to remove the langflow-data/ directory alongside opensearch-data/ and config/

Code Cleanup

- Removed trailing whitespace throughout src/services/flows_service.py

Builds on #1129
Issue

- #1127

Summary

- Propagated LANGFLOW_DATA_PATH to all locations that reference OPENSEARCH_DATA_PATH

CI / Test Infrastructure

- Added langflow-data to the Docker container cleanup command in test-e2e.yml and test-integration.yml, ensuring CI runners start each run with a clean Langflow database
- Added LANGFLOW_DATA_PATH=./langflow-data to frontend/.env.test.example alongside the existing OPENSEARCH_DATA_PATH entry
- Added a langflow-data directory cleanup block to tests/conftest.py that mirrors the existing opensearch-data teardown, so integration tests run against a fresh Langflow DB

Documentation

- Added a LANGFLOW_DATA_PATH row to the Langflow settings table in docs/docs/reference/configuration.mdx, documenting the default value and that flow edits are lost without this volume

TUI — Configuration Fields

- Added a langflow_data_path ConfigField to the Langflow section in config_fields.py with a file picker placeholder and a default of $HOME/.openrag/data/langflow-data

TUI — Env Manager

- Added langflow_data_path field to the EnvConfig dataclass in env_manager.py
- Added "LANGFLOW_DATA_PATH": "langflow_data_path" entry to _env_attr_map() so the variable is loaded from and written to .env files
- Added LANGFLOW_DATA_PATH write in save_env_file() immediately after the OPENSEARCH_DATA_PATH write

TUI — Config Screen

- Added langflow_data_path to SPECIAL_FIELDS in config.py
- Added _render_langflow_data_path() renderer with a "Pick…" directory picker button, matching the _render_opensearch_data_path() pattern
- Added action_pick_langflow_data_path() action implementing the textual-fspicker directory picker flow
- Wired the pick-langflow-data-btn button in on_pressed and added the _langflow_data_pick_callback fallback handler in on_screen_dismissed
Issue

- #1127

Summary

- Fixed Langflow data directory persistence and bind-mount ownership across container restarts

Container & Entrypoint

- Added docker-entrypoint-langflow.sh that runs as root, chowns /app/langflow-data to uid=1000, then drops privileges and execs the main process — mirrors the pattern used by official database images
- Updated Dockerfile.langflow to switch to USER root, copy the entrypoint script, set it as ENTRYPOINT, and simplified the RUN layer
- Added # syntax=docker/dockerfile:1.4 directive to Dockerfile.langflow

Repository & Git Ignore

- Removed langflow-data/.gitkeep and replaced the selective .gitignore pattern (/langflow-data/* + !.gitkeep) with a blanket langflow-data/ ignore entry

Makefile

- Added ensure-langflow-data target that creates the langflow-data bind-mount directory on the host before Docker starts (prevents Docker from creating it as root)
- Added ensure-langflow-data as a prerequisite to dev, dev-cpu, dev-local, dev-local-cpu, dev-local-build-lf, dev-local-build-lf-cpu, dev-branch, dev-branch-cpu, restart-dev, test-ci, and test-ci-local
- Updated factory-reset to fully remove the langflow-data directory (rm -rf langflow-data) instead of only clearing its contents
Issue

- #1127

Summary

- Removed Langflow data directory cleanup from test setup

Test Infrastructure

- Removed the block in conftest.py that deleted the Langflow data directory (LANGFLOW_DATA_PATH) before tests ran, as this cleanup is no longer appropriate given that the Langflow database is now persisted
across container restarts.
Issue

- #1127

Summary

- Persisted Langflow database and flow configurations across container restarts by replacing the blind upsert startup mechanism with a create-only flow seeding approach.

Docker / Container Changes

- Removed the custom docker-entrypoint-langflow.sh entrypoint script, which ran chown as root before dropping to uid=1000; the base Langflow image already owns /app as uid=1000, so the privilege escalation was
unnecessary.
- Simplified Dockerfile.langflow by removing the USER root switch, entrypoint script copy, and the ENTRYPOINT override; the image now runs its default langflow run command directly.
- Updated docker-compose.yml to resolve LANGFLOW_DATABASE_URL from the environment with a SQLite fallback (${LANGFLOW_DATABASE_URL:-sqlite:////app/langflow-data/langflow.db}), allowing operators to substitute
a PostgreSQL URL without rebuilding the image.
- Added LANGFLOW_DATABASE_URL to .env.example with documentation on overriding for production PostgreSQL deployments.

Kubernetes / Helm Changes

- Replaced the LANGFLOW_LOAD_FLOWS_PATH env var with LANGFLOW_CONFIG_DIR in the Langflow Helm deployment template to align with the new startup flow seeding approach.

Backend: Flow Seeding & Reset Detection

- Updated FlowsService.ensure_flows_exist() to return a set[str] of flow type names that were newly created during the current startup, rather than returning None.
- Added handling for unexpected non-404 HTTP status codes when checking whether a flow exists; logs a warning and skips creation to avoid overwriting existing data.
- Updated startup_tasks() in src/main.py to capture the set of newly created flows from ensure_flows_exist().
- Filtered out newly seeded flows from the check_flows_reset() result so that freshly created flows (which match their JSON definition by design) are not incorrectly flagged as having been externally reset.
- Upgraded the log level for ensure_flows_exist() failures from warning to error to better surface critical startup failures.
@mpawlow mpawlow force-pushed the mp/fix/GH-1127-langflow-flow-edits-not-persisted branch from 0656719 to 8566c05 Compare March 19, 2026 21:47
@github-actions github-actions Bot added documentation 📘 Improvements or additions to documentation bug 🔴 Something isn't working. and removed bug 🔴 Something isn't working. documentation 📘 Improvements or additions to documentation labels Mar 19, 2026
@github-actions

This comment has been minimized.

Issue

- #1127

Summary

- Fixed Langflow data directory permissions for CI and E2E test environments

CI Test Pipeline (Makefile)

- Added chmod 777 langflow-data before the test run in both test-ci and test-ci-local targets to ensure the langflow-data directory is world-writable prior to container startup

E2E Test Setup (scripts/setup-e2e.sh)

- Added pre-creation of the langflow-data directory with world-writable permissions (777) before infrastructure starts, ensuring the Langflow container (UID 1000) and the CI runner (UID 1001) can both access it regardless of Docker's :U flag behavior
@github-actions github-actions Bot added documentation 📘 Improvements or additions to documentation bug 🔴 Something isn't working. and removed bug 🔴 Something isn't working. documentation 📘 Improvements or additions to documentation labels Mar 20, 2026
@github-actions

github-actions Bot commented Mar 20, 2026

Copy link
Copy Markdown
Contributor

Build successful! ✅
Deploying docs draft.
Deploy successful! View draft

@github-actions github-actions Bot added bug 🔴 Something isn't working. and removed bug 🔴 Something isn't working. labels Mar 20, 2026
@mpawlow mpawlow requested a review from rodageve March 20, 2026 02:29
@mpawlow

mpawlow commented Mar 20, 2026

Copy link
Copy Markdown
Collaborator Author

Manual Verification Steps

Prerequisites

make factory-reset
  • Confirm:
    • langflow-data/ does not exist (was deleted by factory-reset)

1. Fresh startup — directory creation

  • Goal: Verify langflow-data/ is created by the Makefile and Langflow starts.
make dev-local-cpu
make backend
make frontend
make docling

2. Flow persistence across container restart (core bug fix)

  • Goal: flow edits survive a container restart.
  1. Open http://localhost:7860 in a browser
  2. Open one of the built-in flows (e.g. the chat flow)
  3. Make a visible change — rename a node, change a parameter, add a note
  4. Save the flow
docker compose restart langflow
  • Wait for health check to pass
curl -s http://localhost:7860/health
  • Check:
    • After restart, open the same flow in the Langflow UI
    • Your edit is still present — flow was not reset to the original file

3. Backend restart — flows are preserved without duplication

  • Goal: Verify ensure_flows_exist() creates flows on first start and does not duplicate them on subsequent starts.
  • Restart the backend
Ctrl+C Backend
make backend

4. langflow-data/ bind-mount is active

  • Goal: Verify the database is stored on the host, not inside the container layer.
ls -la langflow-data/
  • Check:
    • langflow.db, secret_key, profile_pictures/ are present on the host
    • Files are owned by uid=1000 (set by the entrypoint chown)
docker compose stop langflow
ls -la langflow-data/   
docker compose start langflow
  • Check:
    • Files are still present after stop/start
    • Langflow resumes with the same database (previous flows still visible in UI)

@rodageve rodageve left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

These changes are mostly to support persisting Langflow data on desktop environments via docker-compose system setup. These changes are low-risk for deployment in SaaS environment where we will be mounting the Langflow config data using Persistent Volumes. Langflow DB data when using Sqlite will also be mounted using Persistent Volumes in SaaS, we're still developing and making changes to the Helm Chart to support this properly.

@github-actions github-actions Bot added the lgtm label Mar 20, 2026

@edwinjosechittilappilly edwinjosechittilappilly left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@mpawlow

mpawlow commented Mar 20, 2026

Copy link
Copy Markdown
Collaborator Author

RE: #1167 (comment)

  • ✅ Successfully verified functionality (sanity check) and file permissions on MacBook
    • (thanks to @Wallgau for doing the verifications and walking through the changes step-by-step)

@mpawlow mpawlow merged commit 0ee0d5e into main Mar 20, 2026
17 checks passed
@github-actions github-actions Bot deleted the mp/fix/GH-1127-langflow-flow-edits-not-persisted branch March 20, 2026 20:29
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

backend 🔷 Issues related to backend services (OpenSearch, Langflow, APIs) bug 🔴 Something isn't working. ci ⬛ CI/CD, build, and infrastructure issues docker frontend 🟨 Issues related to the UI/UX lgtm tests

Projects

None yet

Development

Successfully merging this pull request may close these issues.

[Bug]: Langflow flow edits are not persisting after restart.

4 participants