Repository navigation
fix(cli): end a stalled cloud upload and send it again - #7237
Conversation
There was a problem hiding this comment.
Requesting changes at 3161c0f9.
Blocker: the project zip upload still has no stall bound, and it is the upload this PR is about.
cloud render uploads the project with uploadZipViaDirectUpload({ bytes: archive.buffer, … }) (commands/cloud/render.ts:874). A Uint8Array body takes putBytesOnce, a plain fetch with no stall timer. Only putFileToPresignedUrl (the file-on-disk path, used for large media assets) arms putTimings.stallMs. On a dead link the zip PUT still waits for undici or the OS. Because that PUT is now wrapped in putWithRetry, the hang can repeat up to four times instead of once.
I reproduced this on this head with a probe that copies the PR's own withPlan(["hang", 200]) harness and sets stallMs: 150, but uploads bytes instead of path. It never ends the first try and times out after 8 s. The same probe through the path route passes. All three new tests use path, so nothing covers the bytes route.
Fix: put the same no-progress bound on the bytes path. Either send the buffer through the http(s).request path (for example Readable.from([body]) or chunked slices, so the same arm() runs), or give fetch an AbortController that a progress timer resets. Then add the ["hang", 200] case for a bytes upload. The PR title and body should also say which uploads are covered.
Checked and fine:
- The named test,
ends a try that moves no data and sends the file again, fails whenupload.tsis reverted to the merge base (5 s timeout) and passes on this head. 29/29 pass inupload.test.tsanderrors.test.tson this branch. - Errors below 500 are final, 5xx and network errors are retried, a fresh read stream is opened per try, and the timer is cleared once the response arrives.
Non-blocking:
- The timer is re-armed on
dataevents from the read stream (disk reads), not on bytes leaving the socket. After the last read (end), only one 60 s window covers draining the writable and kernel send buffers plus S3's reply. On a very slow uplink with a large send buffer, that tail could exceed 60 s. A link that is slow but still moving could then be ended near the end, and each retry would restart from byte 0. Re-arming on socket drain/progress, or allowing a longer window afterend, would make "slow but moving links are never ended" hold. sleep()between tries ignoressignal, so an abort waits out the current backoff (up to 4 s). Errors that aren't network errors (for exampleENOENTon the file) are also retried three times.
…ile, watch the socket
|
Fixed at 9653824: the in-memory zip takes the same stalled-try bound (new test 'bounds the project zip, which is sent from memory, the same way'), progress is now read from the socket's accepted bytes so the drain tail is covered, abort cuts the backoff wait, and ENOENT/EACCES are not retried. 31 tests pass; throttled run re-measured: slow link ok 1.7 s, link that dies once recovers in 62 s. |
There was a problem hiding this comment.
Approving at 96538241. The blocker from my review at 3161c0f9 is fixed.
Blocker fixed: uploadZipViaDirectUpload from cloud render passes no fetchImpl, so the in-memory zip now goes through putStreamToPresignedUrl in 64 KB slices and gets the same stall watch. The new test bounds the project zip, which is sent from memory, the same way fails on the previous head's upload.ts (5 s timeout) and passes here. The test from the "fails without the fix" line still fails on the merge base. On this head, 30/30 pass in upload.test.ts and errors.test.ts, and tsc shows no errors in cloud/upload.ts.
Other earlier notes:
- The abort now ends the backoff sleep. The next try then throws immediately on the aborted signal.
ENOENTandEACCESare final.- Stall detection now watches
req.socket.bytesWritteninstead of disk reads. Backpressure stops that counter on a dead link, so a stall is still caught.
Non-blocking:
socket.bytesWrittencounts bytes handed to the socket, including its JS and libuv queues, not bytes the peer has acknowledged. After the last slice is written, a single 60 s window still covers draining the kernel send buffer plus S3's reply. The tail risk I raised earlier for very slow uplinks is smaller but not gone. The PR body's "bytes actually sent" overstates it slightly.- Callers that pass their own
fetchImplkeep thefetchpath without the stall watch, but now get retries. In-repo, those areaudio/analysis/providers/api.ts(boundedFetch) andstems.ts, whose fetch is bounded by its ownAbortSignal.timeout. That's fine as it stands. sleep()never removes its abort listener when the timer fires (at most 3 per upload).
|
Fallow flagged putWithRetry and putStreamToPresignedUrl (CRAP 31.6, cyclomatic 10). 2bcc8d4 splits them: isFinal(), watchStall() and settle() are their own small functions; behavior is unchanged. upload.test.ts and errors.test.ts pass, and fallow audit reports no issues on the changed files. |
There was a problem hiding this comment.
Approving at 2bcc8d4f. Compared with 96538241, which I approved earlier, this is a refactor with no change in behavior.
- Same base: both heads sit on merge base
216395e9. The only new commit is2bcc8d4f, which touches onlycloud/upload.ts. - What moved: the retry-final condition moved into
isFinal(), the socketbytesWritteninterval moved intowatchStall(req)(which returns its clear function), and reading the response moved intosettle(res, sent). The logic, constants, error messages and order of operations are unchanged. The cleanup infinallystill runs before the body is read. - Tests: none changed. 30/30 pass in
upload.test.tsanderrors.test.tson this head, andtscshows no errors incloud/upload.ts.
My earlier non-blocking notes still apply: the tail after the last write, fetchImpl callers keeping the fetch path, and the abort listener in sleep().
…view Adopts heygen-com#7250 (SVG stroke drawing is not a dead stretch), heygen-com#7248 (contract check reads a timeline registered in a local script), heygen-com#7252 (comma-list selectors hidden by authored CSS), heygen-com#7238 (snapshot and preview show a clip on the render frame), heygen-com#7237 (stalled cloud upload retries) and heygen-com#7251 (per-scene SVG id namespacing), with prerequisite heygen-com#6829. Defers heygen-com#7246 (needs the delivery limiter from heygen-com#6940) and heygen-com#7232 (needs the WebKit decode-first pipeline), with conditions in the ledger. Watermark advances to b108f85. Co-authored-by: Miguel Ángel <[email protected]> Co-Authored-By: Claude Opus 5.5 <[email protected]>
Problem
The cloud upload of a project is one whole-file PUT to a presigned URL with no per-try timeout and no retry. On a link that drops or stalls mid-upload the CLI waits for the operating system or the HTTP client to give up (10 minutes and more), then fails and the person loses the whole upload.
Change
Measured (devbox, real constants, local stand-in for S3, 8 MB file)
"Fails in minutes" means 4 stalled tries of 60 s plus 7 s of waits. The whole-file PUT is still restarted from byte 0 on each retry.
Verified
upload.test.tsanderrors.test.ts: 30 pass. Without the change the three new tests fail. oxfmt run; the repo lint hook was not run (no local install).fails without the fix: ends a try that moves no data and sends the file again