Skip to content

fix(syncer): retain Dropbox checkpoints on download failure - #21809

Merged
wangq8 merged 2 commits into
infiniflow:mainfrom
Zhi-Xiang-Guo:fix/dropbox-download-checkpoint
Nov 2, 2026
Merged

wangq8 merged 2 commits into
infiniflow:mainfrom
Zhi-Xiang-Guo:fix/dropbox-download-checkpoint

Conversation

@Zhi-Xiang-Guo

Copy link
Copy Markdown
Contributor

Description

When Dropbox returns a download error, the connector reports a successful batch and checkpoints past the failed file. Resume skips it, and normal sync completion advances the time window past its modification time.

Return the wrapped download error with an empty batch, and commit the session index only after all downloads in the batch succeed. SyncRunner already propagates NextBatch errors without completing the window. Retrying the same session or reopening from the last committed checkpoint replays the uncommitted batch. Successful zero-byte downloads retain the existing skip behavior.

Tradeoff: a persistently unavailable file blocks completion of its sync window until the source is repaired. Previously omitted files require a full sync.

Related Issues

Closes #21808.

Related but separate: #21772 (WebDAV); #21794 (Dropbox timestamp selection, not modified here).

Module(s) Affected

  • internal/syncer/connector/dropbox.go
  • internal/syncer/connector/dropbox_test.go

Testing

Baseline: main@94369fc289e61be2f7e43bd0c65862269207d9c4. macOS arm64, Go 1.27.3. Tested tree ffcca6a5c87d069c994d769b245855e169b5f30d matches remote commit 261ffc3.

  • Before: both new regression subcases fail because a 503 becomes nil error with a checkpoint past the file.
  • After: CGO_ENABLED=0 go test -count=1 -json ./internal/syncer/connector/... passes: 684 top-level tests plus 193 subtests, no failures/skips.
  • CGO_ENABLED=0 go vet ./internal/syncer/connector/..., gofmt, all nine repository file-check functions, and git diff --check pass.
  • Real connector and HTTP transport with httptest cover a failure after an earlier committed batch, a failure within an uncommitted batch, same-session retry and reopening from the last committed checkpoint.
  • bash build.sh --test ./internal/syncer/connector/... was attempted but interrupted while provisioning the missing cl100k table. The pure-Go connector package was tested separately with CGO disabled. Native/full application tests and live Dropbox validation were not run.
  • Full pre-commit was not run; applicable file checks were invoked directly. Remote CI has not yet been observed.

Checklist

  • Targets main; repository instructions reviewed.
  • Regression coverage added.
  • No public API, schema or dependency change.
  • Full build-script/native checks completed.

Additional Notes

AI-assisted investigation, implementation and testing; no human review claim. The runner/watermark consequence was traced in source; executed verification stops at the connector session boundary.

@coderabbitai

coderabbitai Bot commented Oct 31, 2026 •

Copy link
Copy Markdown
Contributor

Review in Change Stack →

Important

Review skipped

Review was skipped as selected files did not have any reviewable changes.

Run configuration
  • Source mode: Auto
  • Repository: Unavailable
  • Branch: Unavailable
  • File: No configuration file from this repository applied
  • Commit: Unavailable
  • Layers: Organization UI
  • Review profile: CHILL
  • Plan: Advanced
  • Run ID: acd02633-87fb-4c69-8058-81cfefedaa68
Commits

Reviewing files that changed from the base of the PR and between 261ffc3 and 4505f55.

You can disable this status message by setting the reviews.review_status to false in the CodeRabbit configuration file.

Use the checkbox below for a quick retry:

  • Trigger review

No actionable comments were generated in the recent review.

Recent review info
Run configuration
  • Source mode: Auto
  • Repository: Unavailable
  • Branch: Unavailable
  • File: No configuration file from this repository applied
  • Commit: Unavailable
  • Layers: Organization UI
  • Review profile: CHILL
  • Plan: Advanced
  • Run ID: 9bea36c3-9014-4e84-86c0-8495cd44055c
Commits

Reviewing files that changed from the base of the PR and between 6ec4e3d and 261ffc3.

Files selected for processing: 2
  • internal/syncer/connector/dropbox.go
  • internal/syncer/connector/dropbox_test.go

Included review availability: This review used your included allowance. Your plan provides up to 10 included reviews per hour; 9 remain after this review.

Walkthrough

Dropbox sync now reports failed downloads instead of advancing past them. It commits the session index only after the batch succeeds. Tests verify retry behavior within the session and from the last committed checkpoint.

Changes: Dropbox sync batch retry

Layer / File(s) Summary
Batch checkpoint and retry behavior
internal/syncer/connector/dropbox.go, internal/syncer/connector/dropbox_test.go
NextBatch now returns an error naming a failed file and leaves the session index uncommitted. Empty downloads remain skipped. Tests cover retrying the session and reopening from the last committed checkpoint.

Estimated code review effort: 2 (Simple) | ~12 minutes

Change: Bug fix · Severity of issue fixed: Medium

Suggested reviewers: haruko386

Merge Risk: Minimal · previous review 261ff

Failed Dropbox downloads remain retryable. No merge-blocking risk was identified.

Pre-merge checks: Passed 4 · Failed 1

Failed Failed checks: 1 warning

Check name Status Explanation Resolution
Docstring Coverage Warning Docstring coverage is 33.33% which is insufficient. The required threshold is 80.00%. Docstring coverage is scoped to functions touched by this diff with cyclomatic complexity of at least 2. Documenta… Write docstrings for the functions missing them to satisfy the coverage threshold.
Passed Passed checks: 4 passed
Check name Status Explanation
Title check Passed The title clearly and concisely describes the main change: preserving Dropbox checkpoints when downloads fail.
Description check Passed The description explains the failure mode, implementation, retry behavior, tradeoff, affected modules, linked issue, tests, and known validation limits. It uses a "### Description" heading instead of …
Linked Issues check Passed Issue #21808 requires Dropbox download failures to avoid acknowledging an incomplete batch and to replay the failed file with preceding uncommitted files. In dropboxSyncSession.NextBatch, the change…
Out of Scope Changes check Passed The pull request changes only internal/syncer/connector/dropbox.go and internal/syncer/connector/dropbox_test.go. The implementation and tests directly support issue #21808 checkpoint and retry be…
Full details: Description check

Explanation

The description explains the failure mode, implementation, retry behavior, tradeoff, affected modules, linked issue, tests, and known validation limits. It uses a "### Description" heading instead of the template's "### Summary" heading, but it provides the required summary information and is otherwise complete.

Full details: Linked Issues check

Explanation

Issue #21808 requires Dropbox download failures to avoid acknowledging an incomplete batch and to replay the failed file with preceding uncommitted files. In dropboxSyncSession.NextBatch, the change uses nextIndex, returns a wrapped error with an empty SyncBatch on download failure, and assigns s.index only after all downloads succeed. The regression test covers batch sizes 1 and 2, same-session retry, reopening from the last checkpoint, and successful zero-byte behavior remains skipped. The changed files are limited to the Dropbox connector and its tests.

Full details: Out of Scope Changes check

Explanation

The pull request changes only internal/syncer/connector/dropbox.go and internal/syncer/connector/dropbox_test.go. The implementation and tests directly support issue #21808 checkpoint and retry behavior. No WebDAV changes, timestamp-selection changes, public API changes, schema changes, or unrelated behavior are present in the reviewed diff.

Full details: Docstring Coverage

Explanation

Docstring coverage is 33.33% which is insufficient. The required threshold is 80.00%. Docstring coverage is scoped to functions touched by this diff with cyclomatic complexity of at least 2. Documentation-only edits and deletions within surviving functions are included; unchanged diff context and fully deleted functions are excluded. Analyzed 3 functions across 2 files.

Resolution

Write docstrings for the functions missing them to satisfy the coverage threshold.

  • Autofix · Keep fixing CodeRabbit findings and required CI, and resolving merge conflicts

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

Share

A rabbit watched the batch proceed,
Then paused when one file failed to feed.
No checkpoint leapt ahead,
The retry found what had not read,
And every file resumed its speed.

@Harsh23Kashyap Harsh23Kashyap left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Approving at 261ffc3, the same commit I verified this morning from a clean checkout (details on #21808): both regression subcases fail on current main - with b.txt returning 503, NextBatch emits a.txt, writes a checkpoint for b.txt, and returns a nil error - and pass on this branch, with the full connector suite green under CGO_ENABLED=0, twice identical. One behavior change for maintainer sign-off: a persistent download failure now blocks window completion instead of checkpointing past it.

@wangq8 wangq8 self-assigned this Nov 2, 2026
@wangq8 wangq8 added the ci Continue Integration label Nov 2, 2026
@wangq8
wangq8 self-requested a review November 2, 2026 02:56
@wangq8

wangq8 commented Nov 2, 2026

Copy link
Copy Markdown
Collaborator

LGTM

@wangq8
wangq8 merged commit ba54d3e into infiniflow:main Nov 2, 2026
6 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

ci Continue Integration

Projects

None yet

Development

Successfully merging this pull request may close these issues.

[Bug]: Dropbox sync checkpoints past failed downloads and can permanently omit files

3 participants