fix: add replace_duplicates option for Google Drive sync by ricofurtado · Pull Request #1674 · langflow-ai/openrag

ricofurtado · 2026-05-25T18:51:59Z

fix: Google Drive sync replaces existing files instead of failing

Problem

When syncing Google Drive via the UI sync button (both the connector-specific /connectors/google_drive/sync and the global /connectors/sync-all endpoints), files that were already indexed would fail with:

File with name 'XXXX' already exists

This happened because ConnectorFileProcessor and LangflowConnectorFileProcessor both gate re-ingestion behind a replace_duplicates flag that defaults to False. The sync code paths were never passing replace_duplicates=True, so every re-sync of a previously indexed file was treated as a duplicate error instead of an update.

The sync_specific_files path (used for manual file selection) already had replace_duplicates wired correctly — the gap was exclusively in the background/button-triggered sync paths.

Changes

src/api/connectors.py — Pass replace_duplicates=True for google_drive in both the connector_sync and sync_all_connectors endpoints, covering the sync_specific_files call (primary path when document IDs are indexed) and the sync_connector_files fallback (filename-based path for older Langflow-ingested files).
src/connectors/service.py — Add replace_duplicates: bool = False parameter to sync_connector_files and thread it through to ConnectorFileProcessor.
src/connectors/langflow_connector_service.py — Add filename_filter and replace_duplicates parameters to sync_connector_files (both were missing, causing a latent TypeError on the fallback path) and thread replace_duplicates through to LangflowConnectorFileProcessor.

Scope

Only Google Drive sync behavior changes. All other connectors continue to receive replace_duplicates=False (the existing default). Manual file uploads via the UI are unaffected — those processors are instantiated separately and their replace_duplicates flag remains user-controlled.

Test plan

Sync a Google Drive connection that already has indexed files — tasks should complete with status indexed or unchanged, no already exists errors
Confirm a modified Google Drive file is re-indexed with updated content after sync
Confirm an unmodified file returns unchanged (hash deduplication still applies)
Verify other connector types (OneDrive, SharePoint, S3, IBM COS) are unaffected
Run pytest tests/unit/ -k connector

Summary by CodeRabbit

New Features
- Google Drive, OneDrive, and SharePoint syncs now replace duplicate files by default for cleaner results.
- Connector syncs expose an explicit "replace duplicates" option to control duplicate handling.
- Connector syncs can be limited by filename filters so only selected files are queued and processed.

coderabbitai · 2026-05-25T18:52:13Z

No actionable comments were generated in the recent review. 🎉

ℹ️ Recent review info

⚙️ Run configuration

Configuration used: Path: .coderabbit.yaml

Review profile: CHILL

Plan: Pro

Run ID: cf8bae49-4c01-4297-9cba-0171e36c54a3

📥 Commits

Reviewing files that changed from the base of the PR and between 0e3926d and 2e1061e.

📒 Files selected for processing (1)

src/api/connectors.py

Walkthrough

Adds a replace_duplicates parameter at the connector service layer, wires it into file processors, extends Langflow service with filename_filter, and has the API layer decide (true only for google_drive, sharepoint, onedrive) and forward that flag into both document-id and filename-filter sync paths.

Changes

Replace Duplicates Parameter Propagation

Layer / File(s)	Summary
Service signatures and wiring `src/connectors/service.py`, `src/connectors/langflow_connector_service.py`	`ConnectorService.sync_connector_files` and `LangflowConnectorService.sync_connector_files` add `replace_duplicates: bool = False` and forward it into their respective file processor constructors.
Langflow filename filtering `src/connectors/langflow_connector_service.py`	`LangflowConnectorService.sync_connector_files` accepts `filename_filter: set = None` and skips files whose `name` is not in the filter during collection.
API decision and forwarding `src/api/connectors.py`	Adds `_connector_sync_should_replace(connector_type)` (true for `google_drive`, `sharepoint`, `onedrive`) and forwards `replace_duplicates=_connector_sync_should_replace(connector_type)` into both document-id (`sync_specific_files`) and filename-filter (`sync_connector_files`) sync code paths in `connector_sync` and `sync_all_connectors`.

Sequence Diagram

sequenceDiagram
  participant Client as API.connector_sync / sync_all_connectors
  participant ConnectorService as ConnectorService.sync_connector_files
  participant LangflowService as LangflowConnectorService.sync_connector_files
  participant FileProcessor as ConnectorFileProcessor / LangflowConnectorFileProcessor

  Client->>ConnectorService: call (replace_duplicates=_connector_sync_should_replace(connector_type))
  ConnectorService->>FileProcessor: instantiate(process file, replace_duplicates flag)
  Client->>LangflowService: call (filename_filter / replace_duplicates)
  LangflowService->>FileProcessor: instantiate(process file, replace_duplicates flag)

🎯 3 (Moderate) | ⏱️ ~20 minutes

Possibly related PRs

langflow-ai/openrag#1663: Introduces and propagates a replace_duplicates flag through connector sync calls with related duplicate-handling plumbing.

Suggested labels

bug

Suggested reviewers

mfortman11

🚥 Pre-merge checks | ✅ 4 | ❌ 1

❌ Failed checks (1 warning)

Check name	Status	Explanation	Resolution
Title check	⚠️ Warning	The title mentions 'Google Drive sync' but the changes also apply to SharePoint and OneDrive, making it incompletely representative of the full scope of changes.	Update the title to reflect that the fix applies to multiple connectors: 'fix: add replace_duplicates option for Google Drive, SharePoint, and OneDrive sync' or use a broader term like 'fix: add replace_duplicates option for connector sync'.

✅ Passed checks (4 passed)

Check name	Status	Explanation
Description Check	✅ Passed	Check skipped - CodeRabbit’s high-level summary is enabled.
Docstring Coverage	✅ Passed	Docstring coverage is 100.00% which is sufficient. The required threshold is 80.00%.
Linked Issues check	✅ Passed	Check skipped because no linked issues were found for this pull request.
Out of Scope Changes check	✅ Passed	Check skipped because no linked issues were found for this pull request.

_{✏️ Tip: You can configure your own custom pre-merge checks in the settings.}

✨ Finishing Touches

📝 Generate docstrings

Create stacked PR
Commit on current branch

🧪 Generate unit tests (beta)

Create PR with unit tests
Commit unit tests in branch google-drive-manual-sync-failing

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

_{Comment @coderabbitai help to get the list of available commands and usage tips.}

coderabbitai

Actionable comments posted: 1

🤖 Prompt for all review comments with AI agents

Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

Inline comments:
In `@src/connectors/langflow_connector_service.py`:
- Around line 232-233: The function accepts filename_filter (and
replace_duplicates) but the Langflow sync path never uses filename_filter;
update the loop in the Langflow sync path that enqueues files so that if
filename_filter is not None you only enqueue files whose filename is a member of
the filename_filter set (e.g., if filename_filter and filename not in
filename_filter: continue), thereby skipping non-matching files before any
enqueue/queue logic and preserving existing replace_duplicates behavior.

🪄 Autofix (Beta)

Fix all unresolved CodeRabbit comments on this PR:

Push a commit to this branch (recommended)
Create a new PR with the fixes

ℹ️ Review info

⚙️ Run configuration

Configuration used: Path: .coderabbit.yaml

Review profile: CHILL

Plan: Pro

Run ID: 96bcd538-ae8a-492a-af34-f02b7e921097

📥 Commits

Reviewing files that changed from the base of the PR and between 76b0cd8 and bd461bf.

📒 Files selected for processing (3)

src/api/connectors.py
src/connectors/langflow_connector_service.py
src/connectors/service.py

lucaseduoli

LGTM

fix: add replace_duplicates option for Google Drive sync

bd461bf

github-actions Bot added the backend 🔷 Issues related to backend services (OpenSearch, Langflow, APIs) label May 25, 2026

github-actions Bot added bug 🔴 Something isn't working. and removed bug 🔴 Something isn't working. labels May 25, 2026

coderabbitai Bot reviewed May 25, 2026

View reviewed changes

Comment thread src/connectors/langflow_connector_service.py

fix: refactor replace_duplicates logic for Google Drive sync

e444776

github-actions Bot added bug 🔴 Something isn't working. and removed bug 🔴 Something isn't working. labels May 25, 2026

fix: add filename filter to sync_connector_files method

0e3926d

github-actions Bot added bug 🔴 Something isn't working. and removed bug 🔴 Something isn't working. labels May 25, 2026

Added check for connector type to sharepoint and onedrive too

2e1061e

lucaseduoli approved these changes May 25, 2026

View reviewed changes

github-actions Bot added lgtm bug 🔴 Something isn't working. and removed bug 🔴 Something isn't working. labels May 25, 2026

ricofurtado merged commit 0961fd1 into main May 25, 2026
16 checks passed

github-actions Bot deleted the google-drive-manual-sync-failing branch May 25, 2026 21:32

coderabbitai Bot mentioned this pull request Jul 3, 2026

fix: error in skip duplicates functionality for folder ingest for connectors (backport of #1941) #2018

Merged

3 tasks

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

fix: add replace_duplicates option for Google Drive sync#1674

fix: add replace_duplicates option for Google Drive sync#1674
ricofurtado merged 4 commits into
mainfrom
google-drive-manual-sync-failing

ricofurtado commented May 25, 2026 •

edited by coderabbitai Bot

Loading

Uh oh!

coderabbitai Bot commented May 25, 2026 •

edited

Loading

❌ Failed checks (1 warning)

Uh oh!

coderabbitai Bot left a comment

Uh oh!

Uh oh!

lucaseduoli left a comment

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

Uh oh!

Conversation

ricofurtado commented May 25, 2026 • edited by coderabbitai Bot Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

fix: Google Drive sync replaces existing files instead of failing

Problem

Changes

Scope

Test plan

Summary by CodeRabbit

Uh oh!

coderabbitai Bot commented May 25, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Walkthrough

Changes

Sequence Diagram

Possibly related PRs

Suggested labels

Suggested reviewers

❌ Failed checks (1 warning)

Uh oh!

coderabbitai Bot left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

lucaseduoli left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

ricofurtado commented May 25, 2026 •

edited by coderabbitai Bot

Loading

coderabbitai Bot commented May 25, 2026 •

edited

Loading