Skip to content

fix(cli): end a stalled cloud upload and send it again - #7237

Merged
miguel-heygen merged 3 commits into
mainfrom
dcloudpro/upload-retry
Nov 5, 2026
Merged

miguel-heygen merged 3 commits into
mainfrom
dcloudpro/upload-retry

Conversation

@miguel-heygen

@miguel-heygen miguel-heygen commented Nov 5, 2026 •

Copy link
Copy Markdown
Collaborator

Problem

The cloud upload of a project is one whole-file PUT to a presigned URL with no per-try timeout and no retry. On a link that drops or stalls mid-upload the CLI waits for the operating system or the HTTP client to give up (10 minutes and more), then fails and the person loses the whole upload.

Change

  • Covers both uploads: the project zip (sent from memory) and large media files (from disk). Both go through one stream path; a buffer is sent in 64 KiB slices so progress is real.
  • Progress is read off the socket (bytes it accepted), not off disk reads, so a slow but moving uplink is not ended while its send buffer drains. A try ends only after 60 s with no byte accepted.
  • A local file error (missing or unreadable file) is not retried, and an abort cuts a backoff wait short.
  • A caller that injects its own fetch (tests) keeps that path unchanged.
  • A try that moves no data for 60 s is ended ("Upload stalled: no data sent for 60 s"). Slow but moving links are never ended.
  • A failed or stalled PUT is sent again to the same presigned URL, up to 3 more times, waiting 1, 2 and 4 s. The URL stays valid and a PUT replaces the object, so the finalize step is unchanged.
  • A refusal below 500 (403 expired signature, 400) is final and not retried; 5xx is retried. Abort signals stop the loop.
  • No server change and no multipart: a retry restarts the whole file, it does not resume.

Measured (devbox, real constants, local stand-in for S3, 8 MB file)

link main this change
slow but moving (throttled reads) ok ok in 1.8 s, 1 PUT, not ended
goes dead after 1 MB, then recovers still waiting at 330 s (killed) ok in 61.2 s, 2 PUTs
dead for good not measured: it waits for the OS fails in 247.6 s, 4 PUTs, "Upload stalled"

"Fails in minutes" means 4 stalled tries of 60 s plus 7 s of waits. The whole-file PUT is still restarted from byte 0 on each retry.

Verified

upload.test.ts and errors.test.ts: 30 pass. Without the change the three new tests fail. oxfmt run; the repo lint hook was not run (no local install).

fails without the fix: ends a try that moves no data and sends the file again

@heygengenesis heygengenesis Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Requesting changes at 3161c0f9.

Blocker: the project zip upload still has no stall bound, and it is the upload this PR is about.

cloud render uploads the project with uploadZipViaDirectUpload({ bytes: archive.buffer, … }) (commands/cloud/render.ts:874). A Uint8Array body takes putBytesOnce, a plain fetch with no stall timer. Only putFileToPresignedUrl (the file-on-disk path, used for large media assets) arms putTimings.stallMs. On a dead link the zip PUT still waits for undici or the OS. Because that PUT is now wrapped in putWithRetry, the hang can repeat up to four times instead of once.

I reproduced this on this head with a probe that copies the PR's own withPlan(["hang", 200]) harness and sets stallMs: 150, but uploads bytes instead of path. It never ends the first try and times out after 8 s. The same probe through the path route passes. All three new tests use path, so nothing covers the bytes route.

Fix: put the same no-progress bound on the bytes path. Either send the buffer through the http(s).request path (for example Readable.from([body]) or chunked slices, so the same arm() runs), or give fetch an AbortController that a progress timer resets. Then add the ["hang", 200] case for a bytes upload. The PR title and body should also say which uploads are covered.

Checked and fine:

  • The named test, ends a try that moves no data and sends the file again, fails when upload.ts is reverted to the merge base (5 s timeout) and passes on this head. 29/29 pass in upload.test.ts and errors.test.ts on this branch.
  • Errors below 500 are final, 5xx and network errors are retried, a fresh read stream is opened per try, and the timer is cleared once the response arrives.

Non-blocking:

  • The timer is re-armed on data events from the read stream (disk reads), not on bytes leaving the socket. After the last read (end), only one 60 s window covers draining the writable and kernel send buffers plus S3's reply. On a very slow uplink with a large send buffer, that tail could exceed 60 s. A link that is slow but still moving could then be ended near the end, and each retry would restart from byte 0. Re-arming on socket drain/progress, or allowing a longer window after end, would make "slow but moving links are never ended" hold.
  • sleep() between tries ignores signal, so an abort waits out the current backoff (up to 4 s). Errors that aren't network errors (for example ENOENT on the file) are also retried three times.

@miguel-heygen

Copy link
Copy Markdown
Collaborator Author

Fixed at 9653824: the in-memory zip takes the same stalled-try bound (new test 'bounds the project zip, which is sent from memory, the same way'), progress is now read from the socket's accepted bytes so the drain tail is covered, abort cuts the backoff wait, and ENOENT/EACCES are not retried. 31 tests pass; throttled run re-measured: slow link ok 1.7 s, link that dies once recovers in 62 s.

@heygengenesis heygengenesis Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Approving at 96538241. The blocker from my review at 3161c0f9 is fixed.

Blocker fixed: uploadZipViaDirectUpload from cloud render passes no fetchImpl, so the in-memory zip now goes through putStreamToPresignedUrl in 64 KB slices and gets the same stall watch. The new test bounds the project zip, which is sent from memory, the same way fails on the previous head's upload.ts (5 s timeout) and passes here. The test from the "fails without the fix" line still fails on the merge base. On this head, 30/30 pass in upload.test.ts and errors.test.ts, and tsc shows no errors in cloud/upload.ts.

Other earlier notes:

  • The abort now ends the backoff sleep. The next try then throws immediately on the aborted signal.
  • ENOENT and EACCES are final.
  • Stall detection now watches req.socket.bytesWritten instead of disk reads. Backpressure stops that counter on a dead link, so a stall is still caught.

Non-blocking:

  • socket.bytesWritten counts bytes handed to the socket, including its JS and libuv queues, not bytes the peer has acknowledged. After the last slice is written, a single 60 s window still covers draining the kernel send buffer plus S3's reply. The tail risk I raised earlier for very slow uplinks is smaller but not gone. The PR body's "bytes actually sent" overstates it slightly.
  • Callers that pass their own fetchImpl keep the fetch path without the stall watch, but now get retries. In-repo, those are audio/analysis/providers/api.ts (boundedFetch) and stems.ts, whose fetch is bounded by its own AbortSignal.timeout. That's fine as it stands.
  • sleep() never removes its abort listener when the timer fires (at most 3 per upload).

@miguel-heygen

Copy link
Copy Markdown
Collaborator Author

Fallow flagged putWithRetry and putStreamToPresignedUrl (CRAP 31.6, cyclomatic 10). 2bcc8d4 splits them: isFinal(), watchStall() and settle() are their own small functions; behavior is unchanged. upload.test.ts and errors.test.ts pass, and fallow audit reports no issues on the changed files.

@heygengenesis heygengenesis Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Approving at 2bcc8d4f. Compared with 96538241, which I approved earlier, this is a refactor with no change in behavior.

  • Same base: both heads sit on merge base 216395e9. The only new commit is 2bcc8d4f, which touches only cloud/upload.ts.
  • What moved: the retry-final condition moved into isFinal(), the socket bytesWritten interval moved into watchStall(req) (which returns its clear function), and reading the response moved into settle(res, sent). The logic, constants, error messages and order of operations are unchanged. The cleanup in finally still runs before the body is read.
  • Tests: none changed. 30/30 pass in upload.test.ts and errors.test.ts on this head, and tsc shows no errors in cloud/upload.ts.

My earlier non-blocking notes still apply: the tail after the last write, fetchImpl callers keeping the fetch path, and the abort listener in sleep().

@miguel-heygen
miguel-heygen added this pull request to the merge queue Nov 5, 2026
Merged via the queue into main with commit f98218a Nov 5, 2026
69 checks passed
@miguel-heygen
miguel-heygen deleted the dcloudpro/upload-retry branch November 5, 2026 17:25
SanHsien added a commit to SanHsien/hyperframes that referenced this pull request Nov 6, 2026
…view

Adopts heygen-com#7250 (SVG stroke drawing is not a dead stretch), heygen-com#7248
(contract check reads a timeline registered in a local script), heygen-com#7252
(comma-list selectors hidden by authored CSS), heygen-com#7238 (snapshot and
preview show a clip on the render frame), heygen-com#7237 (stalled cloud upload
retries) and heygen-com#7251 (per-scene SVG id namespacing), with prerequisite
heygen-com#6829. Defers heygen-com#7246 (needs the delivery limiter from heygen-com#6940) and heygen-com#7232
(needs the WebKit decode-first pipeline), with conditions in the
ledger. Watermark advances to b108f85.

Co-authored-by: Miguel Ángel <[email protected]>
Co-Authored-By: Claude Opus 5.5 <[email protected]>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant