Skip to content

feat: Implements periodic deletion of workflow run logs that exceed t…#23881

Merged
crazywoola merged 3 commits intolanggenius:mainfrom
9527MrLi:mydify/clean-workflow-log
Aug 19, 2025
Merged

feat: Implements periodic deletion of workflow run logs that exceed t…#23881
crazywoola merged 3 commits intolanggenius:mainfrom
9527MrLi:mydify/clean-workflow-log

Conversation

@9527MrLi
Copy link
Copy Markdown
Contributor

…he retention period, with cascade deletion of all historical data associated with workflow_run.

Important

  1. Make sure you have read our contribution guidelines
  2. Ensure there is an associated issue and you have been assigned to it
  3. Use the correct syntax to link this PR: Fixes #<issue number>.

Summary

Close #23879

Main Features

  • Added new clean_workflow_runlogs_precise Celery task for automatic cleanup of expired workflow run logs
  • Supports configurable log retention days (WORKFLOW_LOG_RETENTION_DAYS) and batch processing size
  • Feature can be enabled/disabled via WORKFLOW_LOG_CLEANUP_ENABLED switch

Core Features

  • Complete cascade deletion: Deletes associated data in foreign key dependency order to ensure data consistency
  • Delete message-related tables (MessageFeedback, MessageFile, MessageAnnotation, etc.)
  • Delete workflow-related tables (WorkflowAppLog, WorkflowNodeExecutionModel)
  • Delete session variables and session records
    Finally delete the main WorkflowRun table

Robustness guarantees:

  • Batch processing to avoid memory overflow
  • Built-in retry mechanism (up to 3 retries)
  • Incremental delays on failure (5/10/15 minutes)
  • Transaction protection ensures atomic operations

Monitoring friendly: Provides detailed logging and execution time statistics

Checklist

  • This change requires a documentation update, included: Dify Document
  • I understand that this PR may be closed in case there was no previous discussion or issues. (This doesn't apply to typos!)
  • I've added a test for each change that was introduced, and I tried as much as possible to make a single atomic change.
  • I've updated the documentation accordingly.
  • I ran dev/reformat(backend) and cd web && npx lint-staged(frontend) to appease the lint gods

@dosubot dosubot bot added size:L This PR changes 100-499 lines, ignoring generated files. 🌊 feat:workflow Workflow related stuff. labels Aug 13, 2025
@crazywoola
Copy link
Copy Markdown
Member

I think this should be added into docker/ as well, otherwise it only works for the source code startup.

@Blackoutta
Copy link
Copy Markdown
Contributor

aye, good point mate, we'll add stuff to the docker compose files and.env.example etc.. Forgot about it because we were using helm charts for the deployment ;p

@9527MrLi 9527MrLi force-pushed the mydify/clean-workflow-log branch from 8129cd8 to 4b17ff6 Compare August 14, 2025 01:13
@9527MrLi
Copy link
Copy Markdown
Contributor Author

@crazywoola added!

@9527MrLi 9527MrLi force-pushed the mydify/clean-workflow-log branch from 89f661c to 499f057 Compare August 15, 2025 09:53
@dosubot dosubot bot added size:XXL This PR changes 1000+ lines, ignoring generated files. and removed size:L This PR changes 100-499 lines, ignoring generated files. labels Aug 15, 2025
@9527MrLi 9527MrLi force-pushed the mydify/clean-workflow-log branch from 499f057 to 53826d4 Compare August 15, 2025 10:05
@dosubot dosubot bot added size:L This PR changes 100-499 lines, ignoring generated files. and removed size:XXL This PR changes 1000+ lines, ignoring generated files. labels Aug 15, 2025
@9527MrLi
Copy link
Copy Markdown
Contributor Author

Hi there!
I've resolved the merge conflict and removed the .env.example file as part of the cleanup.

…he retention period, with cascade deletion of all historical data associated with workflow_run.

Co-authored-by: 9527MrLi <[email protected]>
Co-authored-by: 1wangshu <[email protected]>
Co-authored-by: Blackoutta <[email protected]>
@9527MrLi 9527MrLi force-pushed the mydify/clean-workflow-log branch from 60214be to a03f0d4 Compare August 18, 2025 06:04
Copy link
Copy Markdown
Member

@crazywoola crazywoola left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@dosubot dosubot bot added the lgtm This PR has been approved by a maintainer label Aug 19, 2025
@crazywoola
Copy link
Copy Markdown
Member

@9527MrLi FYI, I changed the WORKFLOW_LOG_CLEANUP_ENABLED from True to False in the docker/docker-compose.yaml in order to keep the same behavior as it is. So we might need a documentation of this feature. Would you mind to submit a PR to dify-docs as well?

@crazywoola crazywoola merged commit 7519944 into langgenius:main Aug 19, 2025
9 checks passed
@9527MrLi
Copy link
Copy Markdown
Contributor Author

Alright, I will submit a new PR to dify-docs to include the updates for this release.

@9527MrLi 9527MrLi deleted the mydify/clean-workflow-log branch August 20, 2025 02:51
asukaminato0721 added a commit to asukaminato0721/dify that referenced this pull request Aug 20, 2025
* Restore useLabelStore mistakenly removed in commit 403e2d5 (langgenius#24052)

Co-authored-by: Yongtao Huang <[email protected]>
Co-authored-by: crazywoola <[email protected]>

* chore: synchronize translations (langgenius#24044)

* feat: add testcontainers based tests for metadata service (langgenius#24048)

* feat: add testcontainers based tests for model loadbalancing service (langgenius#24066)

* feat: add select input support to the conversation opener (langgenius#24043)

* feat: add CLAUDE.md for LLM-assisted development guidance (langgenius#23946)

* feat: add Redis SSL/TLS certificate authentication support (langgenius#23624)

* Fix sticky table header transparency with backdrop-filter blur in dark mode (langgenius#23999)

* fix: update first_id logic to use the oldest answer item in chat messages (langgenius#23992)

Co-authored-by: Copilot <[email protected]>
Co-authored-by: crazywoola <[email protected]>

* refactor: improve loading animation and debug panel styles (langgenius#24075)

* fix(oauth): redis compatibility (langgenius#23959)

* feat: enchance prompt and code (langgenius#23633)

Co-authored-by: stream <[email protected]>
Co-authored-by: Stream <[email protected]>
Co-authored-by: Stream <[email protected]>
Co-authored-by: autofix-ci[bot] <114827586+autofix-ci[bot]@users.noreply.github.com>

* fix: return empty list instead of raising exception for qdrant search when score_threshold is 1 (langgenius#24032)

* chore: translate i18n files (langgenius#24081)

Co-authored-by: Stream29 <[email protected]>

* fix: no current code caused code generation show error (langgenius#24086)

* fix(ui): Optimize UI component styles and layouts (langgenius#24090) (langgenius#24092)

* feat: no longer enable auto upgrade when marketplace is disabled (langgenius#24… (langgenius#24101)

* Feature/improve goto anything commands (langgenius#24091)

* chore: translate i18n files (langgenius#24102)

Co-authored-by: crazywoola <[email protected]>

* fix  pg_vector extension requires SUPERUSER, but not available on Huawei Cloud RDS (langgenius#24093)

Co-authored-by: autofix-ci[bot] <114827586+autofix-ci[bot]@users.noreply.github.com>

* Revert "fix  pg_vector extension requires SUPERUSER, but not availabl… (langgenius#24108)

* fix: validate checklist before publishing workflow (langgenius#24104)

* Chore: remove some dead code in experience-enhance-group (langgenius#24110)

Co-authored-by: Yongtao Huang <[email protected]>

* fix: treat default template of code as empty (langgenius#24106)

Co-authored-by: autofix-ci[bot] <114827586+autofix-ci[bot]@users.noreply.github.com>

* Bump pyobvector to 0.2.15 (langgenius#24120)

* Use typing.Literal to replace str places (langgenius#24099)

Co-authored-by: autofix-ci[bot] <114827586+autofix-ci[bot]@users.noreply.github.com>

* feat: add delete avatar functionality with confirmation modal (langgenius#24127)

Co-authored-by: crazywoola <[email protected]>

* chore: translate i18n files (langgenius#24131)

Co-authored-by: crazywoola <[email protected]>

* an example of suppress (langgenius#24136)

* feat: add testcontainers based tests for feature service (langgenius#24026)

* feat: Implements periodic deletion of workflow run logs that exceed t… (langgenius#23881)

Co-authored-by: shiyun.li973792 <[email protected]>
Co-authored-by: 1wangshu <[email protected]>
Co-authored-by: Blackoutta <[email protected]>
Co-authored-by: crazywoola <[email protected]>

* try ast-grep (langgenius#24149)

* fix: correct behaviour of code fix (langgenius#24152)

Co-authored-by: Joel <[email protected]>
Co-authored-by: autofix-ci[bot] <114827586+autofix-ci[bot]@users.noreply.github.com>

* Fix number input in  tool configure form of agent node tool item (langgenius#24154)

* Remove the second `if self.runtime is None:` check (langgenius#24171)

Co-authored-by: Yongtao Huang <[email protected]>

* Fix: correctly match http/https URLs in image upload file (langgenius#24180)

* feat: add testcontainers based tests for model provider service (langgenius#24193)

* Fix: replace `get_builtin_provider` with `get_plugin_provider` (langgenius#24191)

* docs: format all md files (langgenius#24195)

Signed-off-by: yihong0618 <[email protected]>

* hotfix: fix multiple case match syntax (langgenius#24204)

---------

Signed-off-by: yihong0618 <[email protected]>
Co-authored-by: Yongtao Huang <[email protected]>
Co-authored-by: Yongtao Huang <[email protected]>
Co-authored-by: crazywoola <[email protected]>
Co-authored-by: lyzno1 <[email protected]>
Co-authored-by: NeatGuyCoding <[email protected]>
Co-authored-by: Zhehao Peng <[email protected]>
Co-authored-by: -LAN- <[email protected]>
Co-authored-by: Guangdong Liu <[email protected]>
Co-authored-by: Copilot <[email protected]>
Co-authored-by: Wu Tianwei <[email protected]>
Co-authored-by: Maries <[email protected]>
Co-authored-by: Joel <[email protected]>
Co-authored-by: stream <[email protected]>
Co-authored-by: Stream <[email protected]>
Co-authored-by: Stream <[email protected]>
Co-authored-by: autofix-ci[bot] <114827586+autofix-ci[bot]@users.noreply.github.com>
Co-authored-by: Bo Wu <[email protected]>
Co-authored-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com>
Co-authored-by: Stream29 <[email protected]>
Co-authored-by: HyaCinth <[email protected]>
Co-authored-by: Junyan Qin (Chin) <[email protected]>
Co-authored-by: GuanMu <[email protected]>
Co-authored-by: Elvis_LEE <[email protected]>
Co-authored-by: He Wang <[email protected]>
Co-authored-by: crazywoola <[email protected]>
Co-authored-by: 9527MrLi <[email protected]>
Co-authored-by: shiyun.li973792 <[email protected]>
Co-authored-by: 1wangshu <[email protected]>
Co-authored-by: Blackoutta <[email protected]>
Co-authored-by: KVOJJJin <[email protected]>
Co-authored-by: yihong <[email protected]>
hjlarry pushed a commit that referenced this pull request Aug 20, 2025
#23881)

Co-authored-by: shiyun.li973792 <[email protected]>
Co-authored-by: 1wangshu <[email protected]>
Co-authored-by: Blackoutta <[email protected]>
Co-authored-by: crazywoola <[email protected]>
qiaofenlin pushed a commit to qiaofenlin/dify that referenced this pull request Aug 24, 2025
langgenius#23881)

Co-authored-by: shiyun.li973792 <[email protected]>
Co-authored-by: 1wangshu <[email protected]>
Co-authored-by: Blackoutta <[email protected]>
Co-authored-by: crazywoola <[email protected]>
qiqizjl pushed a commit to qiqizjl/dify that referenced this pull request Aug 27, 2025
langgenius#23881)

Co-authored-by: shiyun.li973792 <[email protected]>
Co-authored-by: 1wangshu <[email protected]>
Co-authored-by: Blackoutta <[email protected]>
Co-authored-by: crazywoola <[email protected]>
HarryReidx pushed a commit to HarryReidx/dify that referenced this pull request Sep 1, 2025
langgenius#23881)

Co-authored-by: shiyun.li973792 <[email protected]>
Co-authored-by: 1wangshu <[email protected]>
Co-authored-by: Blackoutta <[email protected]>
Co-authored-by: crazywoola <[email protected]>
(cherry picked from commit 7519944)
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

🌊 feat:workflow Workflow related stuff. lgtm This PR has been approved by a maintainer size:L This PR changes 100-499 lines, ignoring generated files.

Projects

None yet

3 participants