fix: error in skip duplicates functionality for folder ingest for connectors (backport of #1941)#2018
Conversation
…nectors 75616(#1842) (#1941) * fix: enhance duplicate handling and indexing for connector file uploads * style: ruff autofix (auto) * fix: improve duplicate handling in connector sync functionality * chore: trigger CodeRabbit --------- Co-authored-by: autofix-ci[bot] <114827586+autofix-ci[bot]@users.noreply.github.com> (cherry picked from commit effaf02)
|
No actionable comments were generated in the recent review. 🎉 ℹ️ Recent review info⚙️ Run configurationConfiguration used: Path: .coderabbit.yaml Review profile: CHILL Plan: Pro Run ID: 📒 Files selected for processing (1)
💤 Files with no reviewable changes (1)
WalkthroughBackend duplicate classification now returns explicit counts and file lists, and the sync path skips already-duplicated selections. The upload page and duplicate dialog consume the new count field, and tests cover the updated backend and filename-cleaning behavior. ChangesDuplicate-count classification and sync skip
Estimated code review effort: 3 (Moderate) | ~25 minutes Sequence Diagram(s)sequenceDiagram
participant Frontend
participant ConnectorsAPI
participant ClassifyHelper as _classify_connector_duplicates
participant OpenSearch
Frontend->>ConnectorsAPI: connector_sync(file_infos, replace_duplicates=false)
ConnectorsAPI->>ClassifyHelper: classify selected files
ClassifyHelper->>OpenSearch: query existing filenames
OpenSearch-->>ClassifyHelper: matching filenames
ClassifyHelper-->>ConnectorsAPI: duplicate/non-duplicate lists, counts
alt all files are duplicates
ConnectorsAPI-->>Frontend: 200 no_files with duplicate summary
else some non-duplicates
ConnectorsAPI->>ConnectorsAPI: proceed with sync of non-duplicates
end
Possibly related PRs
Suggested reviewers: 🚥 Pre-merge checks | ✅ 5✅ Passed checks (5 passed)
✨ Finishing Touches📝 Generate docstrings
🧪 Generate unit tests (beta)
Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out. Comment |
|
React Doctor found no new issues. 🎉 Reviewed by React Doctor for commit |
There was a problem hiding this comment.
Caution
Some comments are outside the diff and can’t be posted inline due to platform limitations.
⚠️ Outside diff range comments (1)
tests/unit/test_connector_processor_filename_dedupe.py (1)
262-283: 🎯 Functional Correctness | 🟠 Major | ⚡ Quick winDuplicate test function names shadow earlier tests (F811).
Ruff flags both
test_connector_processor_indexes_cleaned_filename(defined again at Line 287, duplicating Line 229) andtest_langflow_connector_processor_uses_cleaned_filename(defined again at Line 431, duplicating Line 262). Since Python module-leveldefbindings overwrite earlier ones with the same name, pytest will only collect/run the later definition in each pair — the earlier test never executes, silently dropping coverage. Given the PR states 3 new backend tests were added, this duplication likely indicates a merge/cherry-pick artifact where the same test was pasted twice with only minor arg differences (e.g.,connector_id_exists=Trueadded in the second copy).Rename one of each pair (or merge them if they're truly duplicates) so both scenarios are actually exercised.
Also applies to: 286-318, 431-453
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the rest with a brief reason, keep changes minimal, and validate. In `@tests/unit/test_connector_processor_filename_dedupe.py` around lines 262 - 283, There are duplicate module-level test function names in this test file, causing Ruff F811 and overwriting the earlier definitions so only the later tests run. Rename one of each duplicated pair, or merge the duplicated cases, for the tests named test_connector_processor_indexes_cleaned_filename and test_langflow_connector_processor_uses_cleaned_filename so both scenarios remain collected by pytest. Keep the unique intent of each test body tied to the corresponding processor/setup helpers like _build_langflow_processor, _wire_langflow_processor, and process_item.Source: Linters/SAST tools
🧹 Nitpick comments (2)
src/connectors/service.py (1)
653-660: 🎯 Functional Correctness | 🔵 Trivial | ⚡ Quick winApply the same
mime_typefallback insync_connector_files.This fix correctly adds
f.get("mime_type")so OneDrive-style metadata (which emits snake_casemime_type, persrc/connectors/onedrive/connector.py) resolves the cleaned filename. However, the parallel cleaning insync_connector_files(Line 462-468) still uses onlyfile_info.get("mimeType") or file_info.get("mimetype"), so files flowing through that path won't pick upmime_typeand can be indexed under an inconsistent name. Consider aligning both call sites.♻️ Align sync_connector_files (Line 462-468)
original_filenames = { file_info["id"]: clean_connector_filename( - file_info["name"], file_info.get("mimeType") or file_info.get("mimetype") + file_info["name"], + file_info.get("mimeType") + or file_info.get("mime_type") + or file_info.get("mimetype"), ) for file_info in files_to_process if "name" in file_info }🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the rest with a brief reason, keep changes minimal, and validate. In `@src/connectors/service.py` around lines 653 - 660, `sync_connector_files` is still missing the snake_case `mime_type` fallback, so its filename cleaning can diverge from the updated logic used for `all_infos`. Update the MIME type lookup in `sync_connector_files` to match the same fallback order as the other call site, using the existing `clean_connector_filename` flow and `file_info` metadata keys so OneDrive-style records resolve consistently.src/api/connectors.py (1)
986-1013: 🚀 Performance & Scalability | 🔵 TrivialFolder selections are expanded twice against the remote provider.
_classify_connector_duplicatesalready callsconnector.list_files()to expand the selected folders (via_expand_selected_connector_files), and thensync_specific_filesexpands the same selection again with anotherconnector.list_files()call. For large folder ingests this doubles the remote provider listing work on every sync. Consider passing the already-expanded file list through tosync_specific_files(or caching the expansion on the connector) to avoid the second round-trip.🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the rest with a brief reason, keep changes minimal, and validate. In `@src/api/connectors.py` around lines 986 - 1013, The folder-selection flow is expanding the same remote selection twice, once in `_classify_connector_duplicates` and again in `sync_specific_files`, causing duplicate `connector.list_files()` work. Update the sync path so `sync_specific_files` can reuse the already-expanded `file_infos` from the duplicate check (or a cached expansion on the connector) instead of re-expanding folders. Use the existing `_classify_connector_duplicates` and `sync_specific_files` call sites to thread the expanded file list through and skip the second remote listing.
🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
Outside diff comments:
In `@tests/unit/test_connector_processor_filename_dedupe.py`:
- Around line 262-283: There are duplicate module-level test function names in
this test file, causing Ruff F811 and overwriting the earlier definitions so
only the later tests run. Rename one of each duplicated pair, or merge the
duplicated cases, for the tests named
test_connector_processor_indexes_cleaned_filename and
test_langflow_connector_processor_uses_cleaned_filename so both scenarios remain
collected by pytest. Keep the unique intent of each test body tied to the
corresponding processor/setup helpers like _build_langflow_processor,
_wire_langflow_processor, and process_item.
---
Nitpick comments:
In `@src/api/connectors.py`:
- Around line 986-1013: The folder-selection flow is expanding the same remote
selection twice, once in `_classify_connector_duplicates` and again in
`sync_specific_files`, causing duplicate `connector.list_files()` work. Update
the sync path so `sync_specific_files` can reuse the already-expanded
`file_infos` from the duplicate check (or a cached expansion on the connector)
instead of re-expanding folders. Use the existing
`_classify_connector_duplicates` and `sync_specific_files` call sites to thread
the expanded file list through and skip the second remote listing.
In `@src/connectors/service.py`:
- Around line 653-660: `sync_connector_files` is still missing the snake_case
`mime_type` fallback, so its filename cleaning can diverge from the updated
logic used for `all_infos`. Update the MIME type lookup in
`sync_connector_files` to match the same fallback order as the other call site,
using the existing `clean_connector_filename` flow and `file_info` metadata keys
so OneDrive-style records resolve consistently.
ℹ️ Review info
⚙️ Run configuration
Configuration used: Path: .coderabbit.yaml
Review profile: CHILL
Plan: Pro
Run ID: 351796b0-0e86-4352-9840-2b63be18f9b2
📒 Files selected for processing (6)
frontend/app/upload/[provider]/page.tsxfrontend/components/duplicate-handling-dialog.tsxsrc/api/connectors.pysrc/connectors/service.pytests/unit/connectors/test_connector_file_type_validation.pytests/unit/test_connector_processor_filename_dedupe.py
The automated merge-from-main on this branch concatenated two tests (test_connector_processor_indexes_cleaned_filename, test_langflow_connector_processor_uses_cleaned_filename) that already existed under both names, tripping ruff F811. Keep one copy of each, preferring the version wired with connector_id_exists=True to match current check_document_exists behavior.
…to release-saas-ga-0.6.2 (#2021) * fix: error in skip duplicates functionality for folder ingest for connectors (backport of #1941) (#2018) * fix: error in skip duplicates functionality for folder ingest for connectors 75616(#1842) (#1941) * fix: enhance duplicate handling and indexing for connector file uploads * style: ruff autofix (auto) * fix: improve duplicate handling in connector sync functionality * chore: trigger CodeRabbit --------- Co-authored-by: autofix-ci[bot] <114827586+autofix-ci[bot]@users.noreply.github.com> (cherry picked from commit effaf02) * style: ruff autofix (auto) * fix: remove duplicate test definitions from main merge The automated merge-from-main on this branch concatenated two tests (test_connector_processor_indexes_cleaned_filename, test_langflow_connector_processor_uses_cleaned_filename) that already existed under both names, tripping ruff F811. Keep one copy of each, preferring the version wired with connector_id_exists=True to match current check_document_exists behavior. --------- Co-authored-by: autofix-ci[bot] <114827586+autofix-ci[bot]@users.noreply.github.com> (cherry picked from commit 286e9db) * fix: Ingestion of buckets not working right after connection is tested - CPD #75506 (#1942) (#2019) * fix: update bucket ingestion messages for clarity and consistency * fix: guard bucket pre-selection against connection ID mismatch The defaults query returns the first S3/COS connection regardless of active status. Only pre-select saved bucket_names when the defaults connection_id matches the current connector.connectionId to avoid seeding the wrong buckets if a stale connection exists alongside the active one. * fix: prevent defaults from overwriting user bucket selections Two issues in the initial-selection effect: 1. hasAppliedInitial was only set when valid buckets were found, leaving the effect live across bucket refreshes and allowing stale defaults to overwrite selections made after a refetch. 2. No guard against the async race where buckets loads before initialSelectedBuckets: user clicks buckets, defaults arrive later and silently overwrite them. Fix: mark hasAppliedInitial unconditionally on first evaluation, and only apply defaults when selectedBuckets is still empty (user hasn't acted yet). --------- (cherry picked from commit f3823b8) Co-authored-by: Claude Sonnet 4.6 <[email protected]> (cherry picked from commit 36f115f) * fix: handle orphan files and return appropriate responses when syncing connectors (#1785) (#2016) * feat: handle orphan files and return appropriate responses when syncing connectors * feat: refactor GoogleDriveOAuth to separate required scopes and add handling for missing optional group scopes in tests * style: ruff autofix (auto) * feat: add handling for skipped files and warnings in task processing and UI components --------- (cherry picked from commit aca865a) Co-authored-by: autofix-ci[bot] <114827586+autofix-ci[bot]@users.noreply.github.com> (cherry picked from commit 79b33f0) * fix: enhance error handling for document processing and add OCR requirement check for image files (#1859) (#2017) * fix: enhance error handling for document processing and add OCR requirement check for image files * style: ruff autofix (auto) * remove clickhouse connect from langflow image (#1801) * chore: fix ci (#1889) * Update processors.py * test fix * fix ci --------- (cherry picked from commit a286581) Co-authored-by: autofix-ci[bot] <114827586+autofix-ci[bot]@users.noreply.github.com> Co-authored-by: Lucas Oliveira <[email protected]> Co-authored-by: Edwin Jose <[email protected]> (cherry picked from commit 4c6acd8) --------- Co-authored-by: autofix-ci[bot] <114827586+autofix-ci[bot]@users.noreply.github.com> Co-authored-by: Claude Sonnet 4.6 <[email protected]> Co-authored-by: Lucas Oliveira <[email protected]> Co-authored-by: Edwin Jose <[email protected]>
…model option (#2031) * fix: remediate image_scan CVEs in backend, frontend, and langflow images (#2000) * fix light color text catergory chips in light mode (#2013) Co-authored-by: Olfa Maslah <[email protected]> * fix: upgraded version of docling (#1995) * changed version of docling * fix logs not being the correct ones * go back to sequential building to save space * add cache to builds * remove single use results from docling manager * revert single use results * change the ray tenant id header * added env * changed cache to hit if no changes were made * fix hash for cache * style: ruff autofix (auto) * fixed coderabbit * style: ruff autofix (auto) * added comment on docling manager * style: ruff autofix (auto) --------- Co-authored-by: autofix-ci[bot] <114827586+autofix-ci[bot]@users.noreply.github.com> * feat: add chunk level page source (#2015) * Changed export docling document to append page number * update result on frontend to append page number * document how to get page number in readmes * Update flows/components/export_docling_document.py Co-authored-by: coderabbitai[bot] <136622811+coderabbitai[bot]@users.noreply.github.com> * Update flows/components/export_docling_document.py Co-authored-by: coderabbitai[bot] <136622811+coderabbitai[bot]@users.noreply.github.com> * fix: apply CodeRabbit auto-fixes Fixed 1 file(s) based on 1 unresolved review comment. Co-authored-by: CodeRabbit <[email protected]> * update component code --------- Co-authored-by: coderabbitai[bot] <136622811+coderabbitai[bot]@users.noreply.github.com> Co-authored-by: CodeRabbit <[email protected]> * fix: error in skip duplicates functionality for folder ingest for connectors (backport of #1941) (#2018) * fix: error in skip duplicates functionality for folder ingest for connectors 75616(#1842) (#1941) * fix: enhance duplicate handling and indexing for connector file uploads * style: ruff autofix (auto) * fix: improve duplicate handling in connector sync functionality * chore: trigger CodeRabbit --------- Co-authored-by: autofix-ci[bot] <114827586+autofix-ci[bot]@users.noreply.github.com> (cherry picked from commit effaf02) * style: ruff autofix (auto) * fix: remove duplicate test definitions from main merge The automated merge-from-main on this branch concatenated two tests (test_connector_processor_indexes_cleaned_filename, test_langflow_connector_processor_uses_cleaned_filename) that already existed under both names, tripping ruff F811. Keep one copy of each, preferring the version wired with connector_id_exists=True to match current check_document_exists behavior. --------- Co-authored-by: autofix-ci[bot] <114827586+autofix-ci[bot]@users.noreply.github.com> * fix: Ingestion of buckets not working right after connection is tested - CPD #75506 (#1942) (#2019) * fix: update bucket ingestion messages for clarity and consistency * fix: guard bucket pre-selection against connection ID mismatch The defaults query returns the first S3/COS connection regardless of active status. Only pre-select saved bucket_names when the defaults connection_id matches the current connector.connectionId to avoid seeding the wrong buckets if a stale connection exists alongside the active one. * fix: prevent defaults from overwriting user bucket selections Two issues in the initial-selection effect: 1. hasAppliedInitial was only set when valid buckets were found, leaving the effect live across bucket refreshes and allowing stale defaults to overwrite selections made after a refetch. 2. No guard against the async race where buckets loads before initialSelectedBuckets: user clicks buckets, defaults arrive later and silently overwrite them. Fix: mark hasAppliedInitial unconditionally on first evaluation, and only apply defaults when selectedBuckets is still empty (user hasn't acted yet). --------- (cherry picked from commit f3823b8) Co-authored-by: Claude Sonnet 4.6 <[email protected]> * fix: handle orphan files and return appropriate responses when syncing connectors (#1785) (#2016) * feat: handle orphan files and return appropriate responses when syncing connectors * feat: refactor GoogleDriveOAuth to separate required scopes and add handling for missing optional group scopes in tests * style: ruff autofix (auto) * feat: add handling for skipped files and warnings in task processing and UI components --------- (cherry picked from commit aca865a) Co-authored-by: autofix-ci[bot] <114827586+autofix-ci[bot]@users.noreply.github.com> * fix: enhance error handling for document processing and add OCR requirement check for image files (#1859) (#2017) * fix: enhance error handling for document processing and add OCR requirement check for image files * style: ruff autofix (auto) * remove clickhouse connect from langflow image (#1801) * chore: fix ci (#1889) * Update processors.py * test fix * fix ci --------- (cherry picked from commit a286581) Co-authored-by: autofix-ci[bot] <114827586+autofix-ci[bot]@users.noreply.github.com> Co-authored-by: Lucas Oliveira <[email protected]> Co-authored-by: Edwin Jose <[email protected]> * fix: bump golang.org/x/net to v0.55.0 to remediate CVE-2026-25680 and CVE-2026-39821 (#2014) * feat: add new design for messages and add inline references (#2020) * update opensearch multimodal and flows to support parser, chunk size and chunk overlap * update backend to support citations and chunk metadata * update frontend to support citations and metadata, implement new design for assistant message * changed the prompt for the agent * changed the prompt for the agent * remove hallucinated chunk references * adjust design of citations and where popover appears * adjusted design for message, popover and cards * fix React Doctor picks * fix: apply CodeRabbit auto-fixes Fixed 5 file(s) based on 4 unresolved review comments. Co-authored-by: CodeRabbit <[email protected]> * style: apply biome auto-fixes [skip ci] * updated colors in ibm mode * fixed react doctor picks * added correct type and validation for function calls * remove style for message when on onboarding * fix message not being w full * remove search query in end of the message * change popup to use shadcn popover * fix coderabbit's suggestion * increase prompt limit --------- Co-authored-by: coderabbitai[bot] <136622811+coderabbitai[bot]@users.noreply.github.com> Co-authored-by: CodeRabbit <[email protected]> Co-authored-by: github-actions[bot] <github-actions[bot]@users.noreply.github.com> * apply new settings on retry (#2010) * fix: preserve HTTPException status in update_docling_preset (#1586) (#2004) * fix: report actual synced connection count in connector_sync (#1547) (#2006) * fix: restore LLM model values in reapply_all_settings fallback (#1587) (#2008) * fix: prevent partial matches for exact token searches (#2003) * fix: prevent partial matches for exact token searches (#1937) * style: ruff autofix (auto) * chore: remove unused json import --------- Co-authored-by: autofix-ci[bot] <114827586+autofix-ci[bot]@users.noreply.github.com> * fix: target agent component when updating chat flow system prompt (#2001) * fix: target langflow agent component for system prompt update instead of provider LLM component Signed-off-by: vchen7629 <[email protected]> * fix: removed unused llm_provider arg from _update_langflow_system_prompt Signed-off-by: vchen7629 <[email protected]> * test: add unit tests for update_chat_flow_system_prompt Signed-off-by: vchen7629 <[email protected]> * style: ruff autofix (auto) * fix(test): resolve flow path relative to repo root in test_langflow_agent_system_prompt Signed-off-by: vchen7629 <[email protected]> * style: ruff autofix (auto) --------- Signed-off-by: vchen7629 <[email protected]> Co-authored-by: autofix-ci[bot] <114827586+autofix-ci[bot]@users.noreply.github.com> * fix: replaced hardcoded docker.io prefix with a env variable prefix for private repo compatibility (#2002) Signed-off-by: vchen7629 <[email protected]> * fix: resolve Pydantic delta serialization warnings in agent streaming (#2005) * fix: updated agent model dump to exclude delta field Signed-off-by: vchen7629 <[email protected]> * test: added regression unit test to verify response field + no warnings * style: ruff autofix (auto) * test: added additional unit test to verify that model_dump without exclusion raises the warning Signed-off-by: vchen7629 <[email protected]> * style: ruff autofix (auto) * test: tighten delta warning assertion to match specific Pydantic error Signed-off-by: vchen7629 <[email protected]> * style: ruff autofix (auto) --------- Signed-off-by: vchen7629 <[email protected]> Co-authored-by: autofix-ci[bot] <114827586+autofix-ci[bot]@users.noreply.github.com> * fix: prevent exception details from leaking in API error responses (#2007) * fix: removed str(e) in exceptions to prevent internals from leaking Signed-off-by: vchen7629 <[email protected]> * style: ruff autofix (auto) * fix: added missing status_code Signed-off-by: vchen7629 <[email protected]> * style: ruff autofix (auto) --------- Signed-off-by: vchen7629 <[email protected]> Co-authored-by: autofix-ci[bot] <114827586+autofix-ci[bot]@users.noreply.github.com> * Changed docling options to include local support and be shown in the Langflow tab * style: ruff autofix (auto) * enable ollama and delete unused file * fix next lint --------- Signed-off-by: vchen7629 <[email protected]> Co-authored-by: Gautham N Pai <[email protected]> Co-authored-by: Wallgau <[email protected]> Co-authored-by: Olfa Maslah <[email protected]> Co-authored-by: autofix-ci[bot] <114827586+autofix-ci[bot]@users.noreply.github.com> Co-authored-by: coderabbitai[bot] <136622811+coderabbitai[bot]@users.noreply.github.com> Co-authored-by: CodeRabbit <[email protected]> Co-authored-by: Rico Furtado <[email protected]> Co-authored-by: Claude Sonnet 4.6 <[email protected]> Co-authored-by: Edwin Jose <[email protected]> Co-authored-by: github-actions[bot] <github-actions[bot]@users.noreply.github.com> Co-authored-by: Mike Fortman <[email protected]> Co-authored-by: hunterxtang <[email protected]> Co-authored-by: NishadA05 <[email protected]> Co-authored-by: Zephyrus <[email protected]>
Backport of #1941 from
release-cpdtomain.Adds
_classify_connector_duplicatesto/connector_sync: expands folder selections and classifies files as duplicate/non-duplicate up front, returning accurate counts to the frontend and skipping only true duplicates instead of the old all-or-nothing check. Frontend now readsduplicate_count/non_duplicate_filesfrom the response instead of inferring "all duplicate" fromduplicateNames.length === totalFiles, and the duplicate-list dialog uses a collision-safe React key for filenames.Added 3 new backend tests for the classification endpoint; fixed them to pass
request=and mock_connector_access_denied(both required bymain's RBAC gate onconnector_sync, added after this PR's original base).Test plan
python3 -m py_compileon touched filesmainbaseline confirmed via clean-checkout comparison)tsc --noEmitclean on touched frontend files🤖 Generated with Claude Code
Summary by CodeRabbit
New Features
Bug Fixes