Skip to content

fix: handle orphan files and return appropriate responses when syncing connectors#1715

Merged
ricofurtado merged 3 commits into
mainfrom
wrong-message-when-deleting-all-files-from-google-drive-and-sync
Jun 2, 2026
Merged

fix: handle orphan files and return appropriate responses when syncing connectors#1715
ricofurtado merged 3 commits into
mainfrom
wrong-message-when-deleting-all-files-from-google-drive-and-sync

Conversation

@ricofurtado

@ricofurtado ricofurtado commented Jun 1, 2026

Copy link
Copy Markdown
Collaborator

l

Summary by CodeRabbit

  • Bug Fixes

    • Google Drive connector treats trashed files/shortcut targets as missing so they are excluded from sync.
  • New Features

    • Sync now reconciles orphaned/stale files before syncing and skips connectors with only deletions.
    • Responses include a clear list of connectors that had only deleted files.
  • Tests

    • Added unit tests covering orphan reconciliation, trashed shortcut handling, and related sync flows.

@github-actions github-actions Bot added frontend 🟨 Issues related to the UI/UX backend 🔷 Issues related to backend services (OpenSearch, Langflow, APIs) tests labels Jun 1, 2026
@coderabbitai

coderabbitai Bot commented Jun 1, 2026

Copy link
Copy Markdown
Contributor

Review Change Stack

No actionable comments were generated in the recent review. 🎉

ℹ️ Recent review info
⚙️ Run configuration

Configuration used: Path: .coderabbit.yaml

Review profile: CHILL

Plan: Pro

Run ID: 98b19732-2818-4e54-84ad-6adce68ef889

📥 Commits

Reviewing files that changed from the base of the PR and between 055ec30 and 2a0cb23.

📒 Files selected for processing (2)
  • src/connectors/google_drive/connector.py
  • tests/unit/connectors/test_google_drive_orphan_detection.py
🚧 Files skipped from review as they are similar to previous changes (1)
  • src/connectors/google_drive/connector.py

Walkthrough

This PR filters trashed/orphaned connector files before syncing (Drive-aware), reconciles and skips orphan IDs, reports connectors that only had deletions, adjusts preview counts, updates the frontend response type, and adds unit tests covering these behaviors.

Changes

Orphan Reconciliation in Connector Sync

Layer / File(s) Summary
Google Drive trashed file filtering
src/connectors/google_drive/connector.py
_get_file_meta_by_id() requests and checks trashed, returns None for trashed/404 items and trashed shortcut targets; _iter_selected_items() skips trashed resolved targets during folder expansion.
Orphan filtering in connector_sync
src/api/connectors.py
connector_sync() builds ids_to_sync, reconciles orphans when max_files is unset, removes orphan IDs, returns no_files if none remain, otherwise calls sync_specific_files with filtered IDs.
Orphan tracking in sync_all_connectors
src/api/connectors.py
sync_all_connectors() initializes deleted_only_connectors, reconciles per connector type, removes orphan IDs from existing_file_ids, records connector types with no remaining files, and returns a no_files response with deleted_only_connectors when appropriate.
Orphan preview accounting
src/api/connectors.py
_preview_orphans_for_connector_type() subtracts computed orphan count from synced_count.
Frontend SyncResponse type update
frontend/app/api/mutations/useSyncConnector.ts
Adds optional deleted_only_connectors?: string[] to the SyncResponse type.
Orphan reconciliation test coverage
tests/unit/api/test_reconcile_orphans_for_connector_type.py, tests/unit/connectors/test_google_drive_orphan_detection.py
Adds JSON helper and tests verifying preview accounting, connector_sync orphan filtering and no_files responses, sync_all_connectors deleted-only behavior, and Google Drive trashed-shortcut behaviors; updates test helper signatures accordingly.

Sequence Diagram(s)

sequenceDiagram
  participant Client
  participant SyncAll
  participant ReconcileOrphans
  participant SyncSpecific
  Client->>SyncAll: POST /sync_all
  SyncAll->>ReconcileOrphans: reconcile_orphans_for_connector_type(existing_file_ids)
  ReconcileOrphans-->>SyncAll: orphan_ids
  SyncAll->>SyncAll: existing_file_ids -= orphan_ids
  alt existing_file_ids empty
    SyncAll-->>Client: 200 { status: "no_files", deleted_only_connectors: [...] }
  else
    SyncAll->>SyncSpecific: sync_specific_files(existing_file_ids)
    SyncSpecific-->>SyncAll: tasks/results
    SyncAll-->>Client: 200 { status: "ok", tasks: [...] }
  end
Loading

Estimated code review effort

🎯 3 (Moderate) | ⏱️ ~20 minutes

Possibly related PRs

  • langflow-ai/openrag#1592: Related prior work adding orphan-id reconciliation primitives and wiring into connector sync flows referenced by this PR.
  • langflow-ai/openrag#1694: Overlapping changes around orphan-reconciliation logic and tests for connector sync behaviors.
  • langflow-ai/openrag#1672: Overlaps in Google Drive connector changes (metadata/trashed handling and related areas).

Suggested reviewers

  • edwinjosechittilappilly
  • lucaseduoli
🚥 Pre-merge checks | ✅ 4 | ❌ 1

❌ Failed checks (1 warning)

Check name Status Explanation Resolution
Docstring Coverage ⚠️ Warning Docstring coverage is 50.00% which is insufficient. The required threshold is 80.00%. Write docstrings for the functions missing them to satisfy the coverage threshold.
✅ Passed checks (4 passed)
Check name Status Explanation
Description Check ✅ Passed Check skipped - CodeRabbit’s high-level summary is enabled.
Title check ✅ Passed The title clearly summarizes the main objective: handling orphan files and returning appropriate sync responses.
Linked Issues check ✅ Passed Check skipped because no linked issues were found for this pull request.
Out of Scope Changes check ✅ Passed Check skipped because no linked issues were found for this pull request.

✏️ Tip: You can configure your own custom pre-merge checks in the settings.

✨ Finishing Touches
📝 Generate docstrings
  • Create stacked PR
  • Commit on current branch
🧪 Generate unit tests (beta)
  • Create PR with unit tests
  • Commit unit tests in branch wrong-message-when-deleting-all-files-from-google-drive-and-sync

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

Comment @coderabbitai help to get the list of available commands and usage tips.

@github-actions github-actions Bot added the bug 🔴 Something isn't working. label Jun 1, 2026

@coderabbitai coderabbitai Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Actionable comments posted: 1

Caution

Some comments are outside the diff and can’t be posted inline due to platform limitations.

⚠️ Outside diff range comments (2)
src/api/connectors.py (2)

699-720: ⚠️ Potential issue | 🟠 Major | ⚡ Quick win

max_files skips orphan cleanup without actually limiting this resync.

In this branch, body.max_files is never passed to reconcile_orphans_for_connector_type() or sync_specific_files(). Setting it only disables orphan filtering, but the endpoint still enqueues every indexed ID, including stale ones.

🐛 Proposed fix
-                if body.max_files is None:
-                    orphan_ids = await reconcile_orphans_for_connector_type(
-                        connector_type=connector_type,
-                        user_id=user.user_id,
-                        connector_service=connector_service,
-                        session_manager=session_manager,
-                        jwt_token=jwt_token,
-                        existing_file_ids=existing_file_ids,
-                        id_field=id_field,
-                    )
-                    if orphan_ids:
-                        orphan_id_set = set(orphan_ids)
-                        ids_to_sync = [fid for fid in existing_file_ids if fid not in orphan_id_set]
+                orphan_ids = await reconcile_orphans_for_connector_type(
+                    connector_type=connector_type,
+                    user_id=user.user_id,
+                    connector_service=connector_service,
+                    session_manager=session_manager,
+                    jwt_token=jwt_token,
+                    existing_file_ids=existing_file_ids,
+                    id_field=id_field,
+                )
+                if orphan_ids:
+                    orphan_id_set = set(orphan_ids)
+                    ids_to_sync = [fid for fid in existing_file_ids if fid not in orphan_id_set]
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

In `@src/api/connectors.py` around lines 699 - 720, The resync currently ignores
body.max_files — pass body.max_files into reconcile_orphans_for_connector_type
so orphan cleanup respects the limit, and ensure ids_to_sync is truncated to
body.max_files (or pass max_files into sync_specific_files) before enqueuing;
update references in this block to use body.max_files when computing ids_to_sync
and when calling reconcile_orphans_for_connector_type or sync_specific_files so
only up to max_files IDs are processed.

699-724: 🛠️ Refactor suggestion | 🟠 Major | 🏗️ Heavy lift

Extract orphan sync planning out of the FastAPI handlers.

The new orphan-filter / deleted-only branching is now duplicated across two routes and is already drifting in behavior. Moving this planning into connector_service or a dedicated helper would keep the handlers as HTTP adapters and centralize one sync policy.

As per coding guidelines, src/api/**/*.py: "No business logic in route handlers."

Also applies to: 1188-1204, 1248-1257

🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

In `@src/api/connectors.py` around lines 699 - 724, The orphan-filtering and
deleted-only branching logic (the reconcile_orphans_for_connector_type call and
the ids_to_sync build-up) should be moved out of the FastAPI handler into
connector_service (or a small helper) so the handler is just an HTTP adapter;
add a method like connector_service.plan_ids_to_sync(existing_file_ids,
max_files, connector_type, user_id, session_manager, jwt_token, id_field) that
encapsulates the current logic (call reconcile_orphans_for_connector_type when
max_files is None, compute orphan_id_set and final ids_to_sync, and return the
ids_to_sync or an empty list/None to indicate no files), then replace the
in-handler block that references reconcile_orphans_for_connector_type,
ids_to_sync, and the subsequent if-not-ids_to_sync response with a simple call
to that new method and handle the no-files response; do the same replacement at
the other duplicated locations (the other handler blocks that currently
duplicate this logic).
🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

Inline comments:
In `@src/connectors/google_drive/connector.py`:
- Around line 347-349: The shortcut check is ineffective because
_resolve_shortcut() never requests the target file's "trashed" field, so
resolved.get("trashed") is always falsy; update _resolve_shortcut to include the
target's trashed attribute when fetching the shortcut target (e.g., add
"trashed" to the fields param used in the files().get or files().list call that
resolves the shortcut) so resolved contains the real trashed value, and apply
the same fix for the other shortcut check site (the similar guard around lines
425-426) so both guards correctly detect trashed targets.

---

Outside diff comments:
In `@src/api/connectors.py`:
- Around line 699-720: The resync currently ignores body.max_files — pass
body.max_files into reconcile_orphans_for_connector_type so orphan cleanup
respects the limit, and ensure ids_to_sync is truncated to body.max_files (or
pass max_files into sync_specific_files) before enqueuing; update references in
this block to use body.max_files when computing ids_to_sync and when calling
reconcile_orphans_for_connector_type or sync_specific_files so only up to
max_files IDs are processed.
- Around line 699-724: The orphan-filtering and deleted-only branching logic
(the reconcile_orphans_for_connector_type call and the ids_to_sync build-up)
should be moved out of the FastAPI handler into connector_service (or a small
helper) so the handler is just an HTTP adapter; add a method like
connector_service.plan_ids_to_sync(existing_file_ids, max_files, connector_type,
user_id, session_manager, jwt_token, id_field) that encapsulates the current
logic (call reconcile_orphans_for_connector_type when max_files is None, compute
orphan_id_set and final ids_to_sync, and return the ids_to_sync or an empty
list/None to indicate no files), then replace the in-handler block that
references reconcile_orphans_for_connector_type, ids_to_sync, and the subsequent
if-not-ids_to_sync response with a simple call to that new method and handle the
no-files response; do the same replacement at the other duplicated locations
(the other handler blocks that currently duplicate this logic).
🪄 Autofix (Beta)

Fix all unresolved CodeRabbit comments on this PR:

  • Push a commit to this branch (recommended)
  • Create a new PR with the fixes

ℹ️ Review info
⚙️ Run configuration

Configuration used: Path: .coderabbit.yaml

Review profile: CHILL

Plan: Pro

Run ID: 71b33e51-d08c-41ab-a2ce-804f30683879

📥 Commits

Reviewing files that changed from the base of the PR and between 8f4e663 and b6e77c0.

📒 Files selected for processing (4)
  • frontend/app/api/mutations/useSyncConnector.ts
  • src/api/connectors.py
  • src/connectors/google_drive/connector.py
  • tests/unit/api/test_reconcile_orphans_for_connector_type.py

Comment thread src/connectors/google_drive/connector.py
@github-actions github-actions Bot added docker bug 🔴 Something isn't working. and removed bug 🔴 Something isn't working. labels Jun 1, 2026
@github-actions github-actions Bot added bug 🔴 Something isn't working. and removed bug 🔴 Something isn't working. labels Jun 1, 2026
@ricofurtado ricofurtado force-pushed the wrong-message-when-deleting-all-files-from-google-drive-and-sync branch from 89287e6 to 055ec30 Compare June 1, 2026 22:08
@github-actions github-actions Bot added bug 🔴 Something isn't working. and removed bug 🔴 Something isn't working. labels Jun 1, 2026
@github-actions github-actions Bot added bug 🔴 Something isn't working. and removed bug 🔴 Something isn't working. labels Jun 1, 2026
@ricofurtado ricofurtado enabled auto-merge (squash) June 2, 2026 02:06
@ricofurtado ricofurtado merged commit b13f7d4 into main Jun 2, 2026
21 checks passed
@github-actions github-actions Bot deleted the wrong-message-when-deleting-all-files-from-google-drive-and-sync branch June 2, 2026 13:13
@github-actions github-actions Bot added the lgtm label Jun 2, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

backend 🔷 Issues related to backend services (OpenSearch, Langflow, APIs) bug 🔴 Something isn't working. docker frontend 🟨 Issues related to the UI/UX lgtm tests

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants