Add cloud server log forwarding via WebSocket by emranemran · Pull Request #548 · daydreamlive/scope

emranemran · 2026-02-27T06:00:43Z

Summary

Forward fal.ai server-side logs to the client through the existing Cloud WebSocket connection, making cloud-side debugging visible locally.

Subprocess stdout captured via Popen with PYTHONUNBUFFERED=1, batched (up to 50 lines), and sent as {"type": "logs"} JSON messages
Client re-emits cloud logs into Python logging via scope.cloud logger — appears in terminal, log file, and Electron LogViewer interleaved with local logs
Thread-safe LogBroadcaster using stdlib queue.Queue to bridge the sync subprocess reader thread and async WebSocket forwarder
Server-side log filtering: noisy loggers (e.g. kafka_publisher) are skipped from WebSocket forwarding while still appearing in fal container stdout. Configurable via CLOUD_LOG_SKIP_LOGGERS env var
Frontend log panel (toggled via StatusBar terminal icon) with filter/copy/clear and visual demarcation of cloud vs local lines
GET /api/v1/logs/tail endpoint for incremental log polling

How it works

Cloud Relay Mode (local backend proxies to fal.ai)

┌──────────┐      ┌──────────────┐      ┌──────────────┐      ┌──────────┐
│ fal.ai   │      │ fal_app.py   │      │ Local Backend│      │ Frontend │
│ Scope    │      │ (WebSocket   │      │ (cloud_conn  │      │ (React)  │
│ Backend  │      │  handler)    │      │  manager)    │      │          │
│(subprocess)     │              │      │              │      │          │
└────┬─────┘      └──────┬───────┘      └──────┬───────┘      └────┬─────┘
     │                   │                     │                   │
     │  stdout line      │                     │                   │
     │──────────────────>│                     │                   │
     │                   │                     │                   │
     │  (filter noisy    │                     │                   │
     │   loggers)        │                     │                   │
     │                   │  WebSocket JSON     │                   │
     │                   │  {"type":"logs",    │                   │
     │                   │   "lines":["..."]}  │                   │
     │                   │────────────────────>│                   │
     │                   │                     │                   │
     │                   │                     │  scope.cloud      │
     │                   │                     │  logger.info(     │
     │                   │                     │  "<line>")        │
     │                   │                     │       │           │
     │                   │                     │       ├──> Terminal (stdout)
     │                   │                     │       ├──> Log file (~/.daydream-scope/logs/)
     │                   │                     │       └──> Electron LogViewer (reads file)
     │                   │                     │                   │
     │                   │                     │  GET /api/v1/     │
     │                   │                     │  logs/tail        │
     │                   │                     │<──────────────────│
     │                   │                     │                   │
     │                   │                     │  {"lines":[...],  │
     │                   │                     │   "offset":1234}  │
     │                   │                     │──────────────────>│
     │                   │                     │                   │
     │                   │                     │                   │  Render in
     │                   │                     │                   │  LogPanel

Direct Cloud Mode (browser connects directly to fal.ai)

┌──────────┐      ┌──────────────┐      ┌──────────┐
│ fal.ai   │      │ fal_app.py   │      │ Frontend │
│ Scope    │      │ (WebSocket   │      │ (React)  │
│ Backend  │      │  handler)    │      │          │
│(subprocess)     │              │      │          │
└────┬─────┘      └──────┬───────┘      └────┬─────┘
     │                   │                   │
     │  stdout lines     │                   │
     │──────────────────>│                   │
     │                   │  WebSocket JSON   │
     │                   │  {"type":"logs",  │
     │                   │   "lines":["..."]}│
     │                   │──────────────────>│
     │                   │                   │
     │                   │                   │  CloudAdapter
     │                   │                   │  .onLogs()
     │                   │                   │  callback
     │                   │                   │
     │                   │                   │  Render in
     │                   │                   │  LogPanel

Key implementation details

Server-side (fal.ai)

fal_app.py: LogBroadcaster uses stdlib queue.Queue (thread-safe) — the subprocess reader thread calls publish(), the async forwarder polls via get_nowait() + asyncio.sleep(0.5)
PYTHONUNBUFFERED=1 set in subprocess env to prevent Python block-buffering stdout when piped
_should_forward_log() filters noisy loggers (default: kafka_publisher) before WebSocket batching — full logs remain on fal container stdout
Configurable via CLOUD_LOG_SKIP_LOGGERS env var

Client-side (local backend)

cloud_connection.py: Routes type: "logs" messages to _handle_cloud_logs(), re-emits via scope.cloud logger (no [CLOUD] prefix — the logger name identifies the source)
app.py: /api/v1/logs/tail endpoint with byte offset tracking

Frontend

cloudAdapter.ts: onLogs() handler for direct cloud mode
useLogStream.ts: Unified hook — WS push (cloud) + polling (local/relay). Detects cloud lines via "- scope.cloud -" logger name
LogPanel.tsx: Bottom panel with All/Errors/Cloud filters, color-coded levels, auto-scroll, copy. Cloud lines have blue left border + subtle background tint for visual demarcation
StatusBar.tsx: Terminal icon + "Logs" label with unread badge

Electron

LogViewer.js + LogViewer.html: - scope.cloud - marker with purple highlighting

Test plan

Deploy to fal.ai, connect in cloud relay mode, verify fal subprocess logs appear locally via scope.cloud logger
Verify kafka_publisher logs do NOT appear locally but DO appear on fal dashboard
curl "http://localhost:8000/api/v1/logs/tail?lines=10" returns recent log lines with offset
Open scope in browser, click "Logs" in status bar, verify log panel streams logs
Verify cloud lines have blue left border + background tint, local lines are plain
Test filter buttons (All / Errors / Cloud) and copy button
Open Electron LogViewer (Ctrl+Shift+L), verify cloud lines highlighted in purple
npm run build succeeds

Forward fal.ai server-side logs to the client through the existing cloud WebSocket connection. Logs are captured from the subprocess stdout, batched, and sent as JSON messages to the client where they are re-emitted into Python logging with a [CLOUD] prefix. This makes cloud-side logs visible in the terminal, log file, and Electron LogViewer — interleaved chronologically with local logs. Also adds a frontend log panel (toggle via StatusBar terminal icon) with filter/copy/clear functionality, and a /api/v1/logs/tail endpoint for efficient incremental log polling. Signed-off-by: emranemran <[email protected]>

github-actions · 2026-03-02T20:54:13Z

🚀 fal.ai Preview Deployment


App ID	`daydream/scope-pr-548--preview`
WebSocket	`wss://fal.run/daydream/scope-pr-548--preview/ws`
Commit	`d7e16a3`

Testing

Connect to this preview deployment by setting the fal endpoint in your client:

FAL_WS_URL=wss://fal.run/daydream/scope-pr-548--preview/ws

🧪 E2E tests will run automatically against this deployment.

github-actions · 2026-03-02T20:58:04Z

✅ E2E Tests passed


Status	passed
fal App	`daydream/scope-pr-548--preview`
Run	View logs

Test Artifacts

Check the workflow run for screenshots.

…tdout LogBroadcaster was using asyncio.Queue which is not thread-safe — publish() from the subprocess reader thread couldn't properly wake up the async forwarder's await q.get(). Switch to stdlib queue.Queue with polling. Also set PYTHONUNBUFFERED=1 in subprocess env to prevent Python's block buffering when stdout is a pipe (lines were stuck in 8KB buffer). Signed-off-by: emranemran <[email protected]>

mjh1

Does the logs panel have any kind of limit on the number of logs? Just wondering for long sessions if it might slow the UI down or anything.

mjh1 · 2026-03-03T17:29:20Z

src/scope/server/cloud_connection.py

        else:
            # Unsolicited message (e.g., notifications)
            logger.debug(f"Received unsolicited message: {msg_type}")
+            msg_type = data.get("type")
+            if msg_type == "logs":
+                self._handle_cloud_logs(data)
+            else:
+                logger.debug(f"Received unsolicited message: {msg_type}")


we're already getting msg_type above, and we don't want this extra "unsolicited message" log. so it could just become an elif then keep the existing else as it was

Suggested change

else:

# Unsolicited message (e.g., notifications)

logger.debug(f"Received unsolicited message: {msg_type}")

msg_type = data.get("type")

if msg_type == "logs":

self._handle_cloud_logs(data)

else:

logger.debug(f"Received unsolicited message: {msg_type}")

elif msg_type == "logs":

self._handle_cloud_logs(data)

else:

# Unsolicited message (e.g., notifications)

logger.debug(f"Received unsolicited message: {msg_type}")

mjh1 · 2026-03-03T17:42:09Z

I've made a fix so that we don't fetch duplicate logs over and over: 81f3819 (i have to thank claude for spotting that one). It didn't affect the UI too much but it threw off the new log counter thing (showing 99+ when really the logs weren't changing).

Signed-off-by: Max Holland <[email protected]>

livepeer-tessa · 2026-03-03T18:01:10Z

Good catch to ask about! Yes, there's a limit — useLogStream.ts caps at MAX_LOG_LINES = 2000. When exceeded, it keeps the most recent 2000 and discards older entries:

const combined = [...prev, ...parsed];
return combined.length > MAX_LOG_LINES
  ? combined.slice(-MAX_LOG_LINES)
  : combined;

So long sessions shouldn't cause UI slowdown or memory buildup.

Skip scope.server.kafka_publisher INFO lines (heartbeats, session events) from being forwarded to the client via WebSocket. These are infrastructure bookkeeping with no debugging value to the user. Full logs remain on fal container stdout. Errors/warnings from skipped loggers are still forwarded. Configurable via CLOUD_LOG_SKIP_LOGGERS env var to add more loggers. Also clean up duplicate logger.debug line in cloud_connection._handle_message. Signed-off-by: emranemran <[email protected]>

Remove redundant [CLOUD] prefix from cloud log lines — the scope.cloud logger name already identifies them. Update frontend and Electron to detect cloud lines via "- scope.cloud -" instead. Add blue left border and subtle background tint to cloud lines in LogPanel for clear visual separation. Add "Logs" label next to terminal icon in StatusBar. Signed-off-by: emranemran <[email protected]>

…t error Logger names (cloud_connection, cloud_webrtc_client, cloud_track, etc.) already identify the source. The frontend log panel now uses visual demarcation (blue border + CLOUD badge) instead of text prefixes. Also moves queue import to module level in fal_app.py to fix ruff F821. Signed-off-by: emranemran <[email protected]>

Signed-off-by: emranemran <[email protected]>

emranemran force-pushed the emran/cloud-logs branch from 9485470 to 4c3e94e Compare March 2, 2026 20:37

mjh1 reviewed Mar 3, 2026

View reviewed changes

fix: don't return duplicate logs

81f3819

Signed-off-by: Max Holland <[email protected]>

emranemran added 2 commits March 3, 2026 13:32

emranemran requested a review from leszko March 4, 2026 05:59

emranemran added 2 commits March 3, 2026 22:13

Fix ruff format: collapse single-line log statements

d7e16a3

Signed-off-by: emranemran <[email protected]>

mjh1 approved these changes Mar 4, 2026

View reviewed changes

emranemran merged commit e1d532a into main Mar 4, 2026
8 checks passed

mjh1 deleted the emran/cloud-logs branch March 11, 2026 12:07

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Add cloud server log forwarding via WebSocket#548

Add cloud server log forwarding via WebSocket#548
emranemran merged 7 commits intomainfrom
emran/cloud-logs

emranemran commented Feb 27, 2026 •

edited

Loading

Uh oh!

github-actions bot commented Mar 2, 2026 •

edited

Loading

Uh oh!

github-actions bot commented Mar 2, 2026 •

edited

Loading

Uh oh!

mjh1 left a comment

Uh oh!

mjh1 Mar 3, 2026

Uh oh!

mjh1 commented Mar 3, 2026 •

edited

Loading

Uh oh!

livepeer-tessa commented Mar 3, 2026

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

Conversation

emranemran commented Feb 27, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Summary

How it works

Cloud Relay Mode (local backend proxies to fal.ai)

Direct Cloud Mode (browser connects directly to fal.ai)

Key implementation details

Server-side (fal.ai)

Client-side (local backend)

Frontend

Electron

Test plan

Uh oh!

github-actions bot commented Mar 2, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

🚀 fal.ai Preview Deployment

Testing

Uh oh!

github-actions bot commented Mar 2, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

✅ E2E Tests passed

Test Artifacts

Uh oh!

mjh1 left a comment

Choose a reason for hiding this comment

Uh oh!

mjh1 Mar 3, 2026

Choose a reason for hiding this comment

Uh oh!

mjh1 commented Mar 3, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

livepeer-tessa commented Mar 3, 2026

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

emranemran commented Feb 27, 2026 •

edited

Loading

github-actions bot commented Mar 2, 2026 •

edited

Loading

github-actions bot commented Mar 2, 2026 •

edited

Loading

mjh1 commented Mar 3, 2026 •

edited

Loading