Skip to content

fix: use CDN upload to avoid WebSocket timeout#572

Merged
emranemran merged 8 commits intomainfrom
fix/large-asset-upload-websocket-timeout
Mar 4, 2026
Merged

fix: use CDN upload to avoid WebSocket timeout#572
emranemran merged 8 commits intomainfrom
fix/large-asset-upload-websocket-timeout

Conversation

@mjh1
Copy link
Copy Markdown
Contributor

@mjh1 mjh1 commented Mar 3, 2026

Problem

When uploading reference images larger than ~3MB in cloud mode, the WebSocket connection fails with code 1006 (abnormal closure) after approximately 2 seconds.

Logs observed:

2026-02-25 13:14:17,593 - scope.server.cloud_connection - INFO - [CLOUD] API request: POST /api/v1/assets?filename=image.jpg
2026-02-25 13:14:19,098 - scope.server.cloud_connection - WARNING - Cloud WebSocket closed (code=1006, reason=None)

Root Cause

The upload flow was:

  1. Read file → base64 encode (~33% size increase)
  2. Wrap in JSON: {"_base64_content": "<4MB+ string>", ...}
  3. Send entire JSON through WebSocket

For a 3MB image, this creates a ~4MB+ WebSocket message. The fal.run WebSocket infrastructure appears to have message size limits or processing timeouts that cause connection drops for large payloads.

Solution

  1. Upload directly to fal CDN via REST API (token-based auth)
  2. Get CDN URL (e.g., https://v3.fal.media/files/...)
  3. Send only the URL through WebSocket: {"_cdn_url": "...", ...}
  4. Cloud runner downloads from CDN and processes

Changes:

  • cloud_proxy.py: Added _upload_to_fal_cdn() function
  • fal_app.py: Handle _cdn_url payload in addition to _base64_content

Testing

  1. Upload large image (>3MB) in cloud mode → should now work (uses CDN)
  2. Verify thumbnails still work locally (local copy is saved regardless)

livepeer-robot and others added 8 commits February 26, 2026 18:11
Fixes #545

The hardcoded absolute import 'from pipelines.wan2_1.lora.utils' was
causing ModuleNotFoundError when loading LoRA adapters. This module
path doesn't exist - it should be 'scope.core.pipelines.wan2_1.lora.utils'.

Changed to use relative imports consistent with the rest of the file,
and moved the import to the top-level imports block.

Signed-off-by: livepeer-robot <[email protected]>
When uploading reference images >2MB in cloud mode, the base64-encoded
payload was causing WebSocket connection failures (code 1006) after ~2s.

The root cause: Large files (3MB+) become ~4MB+ when base64-encoded,
exceeding fal.run WebSocket infrastructure limits.

Fix:
- Files >2MB: Upload directly to fal CDN via REST API, then pass URL
  through WebSocket for the cloud runner to fetch
- Files <2MB: Keep existing base64 approach (faster, single roundtrip)

Server-side (fal_app.py) now handles both:
- _base64_content: Decode and forward (existing behavior)
- _cdn_url: Download from CDN and forward (new)

Fixes WebSocket closure when uploading large reference images.

Signed-off-by: livepeer-robot <[email protected]>
Signed-off-by: livepeer-robot <[email protected]>
Signed-off-by: Max Holland <[email protected]>
@github-actions
Copy link
Copy Markdown
Contributor

github-actions bot commented Mar 3, 2026

🚀 fal.ai Preview Deployment

App ID daydream/scope-pr-572--preview
WebSocket wss://fal.run/daydream/scope-pr-572--preview/ws
Commit a257b00

Testing

Connect to this preview deployment by setting the fal endpoint in your client:

FAL_WS_URL=wss://fal.run/daydream/scope-pr-572--preview/ws

🧪 E2E tests will run automatically against this deployment.

@github-actions
Copy link
Copy Markdown
Contributor

github-actions bot commented Mar 3, 2026

✅ E2E Tests passed

Status passed
fal App daydream/scope-pr-572--preview
Run View logs

Test Artifacts

Check the workflow run for screenshots.

@mjh1 mjh1 changed the title fix: use CDN upload for to avoid WebSocket timeout fix: use CDN upload to avoid WebSocket timeout Mar 3, 2026
@emranemran emranemran merged commit be2f8d1 into main Mar 4, 2026
17 checks passed
@mjh1 mjh1 deleted the fix/large-asset-upload-websocket-timeout branch March 4, 2026 13:57
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants