Skip to content

Fix content-list duplicate detection - #278

Merged
LarFii merged 3 commits into
HKUDS:mainfrom
FU-max-boop:fix/defer-content-list-status
May 21, 2026
Merged

Fix content-list duplicate detection#278
LarFii merged 3 commits into
HKUDS:mainfrom
FU-max-boop:fix/defer-content-list-status

Conversation

@FU-max-boop

@FU-max-boop FU-max-boop commented May 17, 2026

Copy link
Copy Markdown
Contributor

Summary

  • defer doc_status creation for text-bearing insert_content_list calls until after LightRAG ainsert runs
  • keep early doc_status creation for multimodal-only content that does not call ainsert
  • add regression coverage for both paths

Fixes #277

Tests

  • PYTHONPATH=. pytest tests/test_insert_content_list.py -q
  • PYTHONPATH=. pytest tests/testparser_wiring.py tests/testparser_kwargs.py tests/test_insert_content_list.py -q
  • python -m ruff check raganything/processor.py tests/test_insert_content_list.py
  • python -m compileall raganything/processor.py tests/test_insert_content_list.py
  • git diff --check

@Walkerdine

Walkerdine commented May 19, 2026

Copy link
Copy Markdown

It looks like process_document_complete also has the same issue

@FU-max-boop

FU-max-boop commented May 20, 2026

Copy link
Copy Markdown
Contributor Author

Good catch, thank you. process_document_complete had the same pre-registration path.

I pushed 0dddfa7 to apply the same deferral there: parsed documents with text now let LightRAG create the initial doc_status during ainsert, while multimodal-only parsed documents still create the status record up front because they never call ainsert.

Added regression coverage for both parsed-document paths and re-ran:

  • PYTHONPATH=. pytest tests/test_insert_content_list.py tests/testparser_wiring.py tests/testparser_kwargs.py -q
  • python -m ruff check raganything/processor.py tests/test_insert_content_list.py
  • python -m ruff format --check raganything/processor.py tests/test_insert_content_list.py
  • python -m compileall raganything/processor.py tests/test_insert_content_list.py
  • git diff --check

@LarFii
LarFii merged commit e8c0081 into HKUDS:main May 21, 2026
1 check passed
This was referenced Aug 11, 2026
Closed
Merged
LarFii pushed a commit that referenced this pull request Aug 13, 2026
@qianheng-aws
fix(processor): defer doc_status creation in process_document_complet…
490baa4 
…e_lightrag_api (#334)

process_document_complete_lightrag_api() pre-registered the content-based doc_id with HANDLING before handing the text to LightRAG, so LightRAG's dedup (filter_keys over doc_status) treated the fresh insert as a historical duplicate and skipped indexing entirely: the document reached processed status with zero chunks and zero entities written.

Only create the record up front for multimodal-only content that never reaches ainsert(), mirroring the fix #278 applied to insert_content_list. Also seed created_at with a real timestamp instead of "" in the doc-pre-… bookkeeping records — backends that map created_at as a date reject the empty string and the bulk upsert drops the write silently.

Fixes #328. Same mechanism as #277 / #278.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

No reviews

Assignees

No one assigned

Labels

None yet

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

[Bug]: Duplicate document detection on unique items when using insert_content_list

3 participants

@FU-max-boop @Walkerdine @LarFii