docs(coding-agents): Devin Desktop application session tracing (TH-8294) - #921

Summary

TH-8294, decision N-10: Cascade hook recipe only. Adds a devin_desktop adapter for the shared hook-to-OTLP emitter (#912), a user-level ~/.codeium/windsurf/hooks.json, an installer, and the page integrations/coding-agents/devin-desktop plus its nav entry (after // coding-agents:devin-desktop).

Cascade session (trajectory_id, model label) > Cascade turn (execution_id) > read_code / write_code / run_command / <mcp_tool_name>

• Source: https://docs.devin.ai/desktop/cascade/hooks (re-read 2026-10-15). Registered: pre_user_prompt (turn start), post_read_code, post_write_code, post_run_command, post_mcp_tool_use, post_cascade_response (turn end). show_output: false.
• Every command is the subshell-guarded form (python3 -B "$HOME/.codeium/windsurf/fi-hook-to-otlp/emit.py" devin_desktop <event> 2>/dev/null || true). Cascade runs command via bash -c with the event on stdin and treats exit 2 from a pre-hook as "block"; python3 exits 2 when the script is missing. The guard makes every hook exit 0 with empty stdout even when the emitter or python3 is missing, so pre_user_prompt never blocks. Exit 0 is documented as "action proceeds normally".
• Not registered or mapped: the other pre-hooks (blocking, no id to pair), post_setup_worktree, and post_cascade_response_with_transcript. Transcript paths are never opened (tested with a canary and spies on open, io.open and os.open).
• model_name is kept as llm.model_name and devin_desktop.model_label, documented as a display label, not a stable id ("Unknown" dropped).
• post_run_command spans keep status unset (the hook has no exit code). Read, write and MCP post-hooks fire after a successful action and are OK.
• Cascade has no session hook, so pre_user_prompt also sends a SESSION UPDATE (exports nothing) so the core sweeps state files of old conversations after 24 h (tested).

Deliberate deviation from PRD R-03 / journey 1: post_cascade_response is registered. R-03 and journey 1 say it is not. It is Cascade's only usable turn-end hook: the other one, post_cascade_response_with_transcript, writes the whole conversation to disk and is never registered. The PRD's own outcome (one execution_id turn span) needs a turn end; without the hook, turns close only as incomplete at the next prompt and the last turn of a conversation is never exported. With content capture off the emitter never exports the response markdown it receives on stdin, so R-03's boundary ("response markdown absent") holds. Re-probed at this head: the response marker is 0 in the OTLP bodies, the stored spans and the state files.

Vendor update since the PRD: Devin Local now has a first-party hooks page (https://docs.devin.ai/cli/extensibility/hooks/lifecycle-hooks, shared with Devin CLI). The N-10 follow-up is unblocked; this PR stays Cascade-only.

Review fixes (2026-10-15)

• Fail-open hook commands (subshell form above). test_devin_desktop_hooks_config runs every shipped command through /bin/sh and bash, with the payload on stdin and as a trailing heredoc: emitter missing and python3 missing give exit 0 and empty stdout; after install.sh each hook exits 0, prints nothing and exports its span. The un-parenthesised ... || true form fails the heredoc case (checked). The page's JSON block is tested to equal hooks.json.
• Error text and paths: the command's cwd is no longer captured; a test puts a path with a user name in an MCP error and a command line and asserts it is absent by default, and kept as /Users/<user>/... with content on.
• Dropped devin_desktop.action / devin_desktop.execution_id (duplicated coding_agent.hook.event / coding_agent.turn.id).
• Page: Uninstall section, platform row (macOS and Linux; Windows not supported or tested), "may not" inherit the shell env and shell FI_* wins over the env file, troubleshooting via FI_HOOK_LOG_FILE instead of show_output, dropped the unsourced "failing system hook" claim, transcript hook named as the excluded turn-end hook, opt-in path and state-file wording matched to the core, common fields stated once under the capture table, **Term**: style.
• Merged the current foundation head 6513a26 (merge commit, no force-push) and re-ran tests and E2E on it.

Recheck fixes (2026-10-15, round 2)

• Merged the foundation head 9c6acd6 (quiet adapter-load failure, user-name-only home-path scrub) with a merge commit, no force-push. Tests and E2E re-run on it (below).
• Troubleshooting step 1: "no lines" now also covers the emitter finding no keys (env file missing or misnamed, keys only in the shell) or FI_HOOK_DISABLED=1, and says the turn start alone never logs.
• Troubleshooting step 2: the manual run says it sends one test trace (session fi-hook-test, one read_code span) and offers a FI_HOOK_DISABLED=1 prefix to check the emitter without sending. FI_HOOK_DUMP_FILE is not offered: it is written in the send path and does not stop the send.
• Attribute wording: the common fields are on every tool and turn span; the Cascade session span has no turn id and carries the model as llm.model_name (matches the stored spans).
• State sweep wording: it runs at the next prompt. Uninstall now deletes the state directory (${TMPDIR:-/tmp}/fi-hook-to-otlp-$(id -u)/devin_desktop) and, if no other app uses it, ~/.config/fi-hook-to-otlp/env; install.sh prints the same steps.
• New tests: the page's manual-run command (disabled: nothing sent, logged or written; enabled: exactly that test trace and one http=200 line), no keys or FI_HOOK_DISABLED=1 log nothing, the page's Uninstall block removes the emitter, the TMPDIR state dir and the env file, and the common fields are on tool and turn spans but not on the session root. Each page-backed test fails when its page line is removed (checked).

How to run

cd examples/coding-agents/hook-to-otlp && python3 -B -m unittest discover -s tests -t .   # 82 tests on this head
examples/coding-agents/devin-desktop/install.sh    # ~/.codeium/windsurf/fi-hook-to-otlp + hooks.json
FI_API_KEY=... FI_SECRET_KEY=... FI_BASE_URL=... examples/coding-agents/devin-desktop/run_e2e.sh   # runs the hooks.json commands via bash -c

E2E evidence

• Devin Desktop 3.8.20 (Devin-darwin-arm64-3.8.20.zip, sha256 f6354300f59e1a45b83a11d4b40a915cf9e3a75c4304564ea8967c9b4ae5cd80, commit 2d9020110aa9, language server 1.48.2) downloaded and inspected, not launched. Cascade requires a Devin sign-in, and a GUI launch would write outside an isolated HOME. The bundled language server carries the documented input fields as JSON tags (agent_action_name, tool_info, trajectory_id, execution_id, model_name) and the .devin/hooks.json / .windsurf/hooks.json paths. The bundled agent binary contains every Cascade hook name.
• Doc-shaped payloads, run through the hooks.json commands exactly as Cascade runs them (bash -c, payload on stdin), into local Future AGI at a0bf68e (core 9c6acd6), project th8103-coding-devin-desktop-r3. fi_verify PASS, trace 408aeb0b-1cba-02fd-c058-26626a9ccfd3, 8 spans: Cascade session (model Claude Sonnet 4) > turn 1 (read_code, run_command, write_code, list_commits) and turn 2 (run_command). Stored status: read/write/MCP OK, run_command UNSET. Every hook exited 0; the 7 span-producing hooks logged http=200 and pre_user_prompt logged nothing. Earlier run at 653ec9e: trace 3cf4a2af.
• Opt-in run at 653ec9e (FI_HOOK_CAPTURE_CONTENT=1, project th8103-coding-devin-desktop-content): fi_verify PASS, trace f5743450-31c6-5d70-6887-4d918a0e05f9. On core 9c6acd6 the opt-in path scrubbing is covered by the unit tests.
• npm run build exit 0, npm run audit-links 0 broken content links.

Privacy proof

• Default (trace 408aeb0b, this head): prompt, tool-input, tool-output and response markers, the transcript and worktree canaries, the local user name, the host name, /Users/, /home/, /tmp/ and input.value/output.value are all 0 in both the stored spans and the OTLP bodies as sent. The four markers are also 0 in the state files.
• Opt-in (trace f5743450): prompt, tool-input and response markers present; the read path is stored as /Users/<user>/project/README.md; the fixture user name, transcript and worktree canaries and the local user name are 0.

Verification (2026-10-20)

How others support Devin Desktop/Windsurf: only Cascade hooks are a tracing path (KiroGraph uses post_cascade_response). Braintrust has an MCP connection only; Arize's harness tracing lists Devin CLI, not Desktop; Langfuse, LangSmith, Datadog, Honeycomb, SigNoz, Laminar, Opik, Helicone, PostHog and AgentOps have no page. Sources are on the page.

• Real: Devin Desktop 3.8.20 build inspected; install.sh into an isolated HOME; installed hooks.json commands run via bash -c, all exit 0, empty stdout. Project th8103-livefill-th8294, trace 7a6a0978-c3cd-0dba-4f92-f77d76132f88 (8 spans, 2 turns), fi_verify PASS. Privacy scan: 0 hits for prompt/command/MCP-arg/response markers, /Users/, user and host name.
• Stand-in: inputs are the tool_info examples from the Cascade Hooks page (no signed-in Cascade run). Unverified in real Cascade: loading of ~/.codeium/windsurf/hooks.json, real post_mcp_tool_use payload, post_cascade_response on cancel, one trace per real trajectory_id.
• Pushed c9c1e56; docs build + audit-links pass.

Not available / limitations

No tokens, cost, model-request spans, tool durations, tool call ids, or command exit codes (not in Cascade hook input). No session start/end hook; the root is re-sent with every span and the next prompt deletes per-conversation state unused for 24 h (Uninstall deletes the rest). If Cascade skips post_cascade_response for a turn (e.g. a cancel), the core closes that turn as incomplete when the next prompt starts, so its tools keep their parent (tested). The last turn of a conversation still needs the hook, since Cascade has no session-end hook. macOS and Linux only; Windows would run the bash-syntax command via powershell -Command and is not supported or tested. Devin Local and hosted Devin are not covered.

🤖 Generated with Claude Code