Skip to content

Commit f4c9b77

Browse files
zigriversclaude
andauthored
feat(opencode): make local turns survive, and make failures visible (#48)
Investigation of "OpenCode stops mid-task on a local model and I have to type continue", grounded in OpenCode's own session database (58k assistant turns), its log, the meter's telemetry, and a throwaway OpenAI-compatible endpoint that recorded exactly what OpenCode puts on the wire. The stalls were overwhelmingly the endpoint dying, not OpenCode losing its place. Turns that never completed over 7 days: glm-meter 0.9%, local-orch 0.0%, local-dev 38.1%. By hour, 2026-08-25 00:00-04:00 shows 40 local turns and 40 failures -- exactly the M2 wedge window. Historically, 5,167 of 5,430 local-review stream errors were Bad Gateway clustered on Aug 2-6, the M2 outage that went unreported for four days. What made it look like an OpenCode fault is that it fails silently: three retries, an AI_RetryError line in its log, then the turn ends with nothing on screen. Changes, each tied to something measured: - limit.output 32000 (was 16384, set earlier today). local-orch has produced 32,000-token turns and been truncated at exactly that ceiling three times; 2 of the 34 recorded "continue" nudges followed such a turn. 16384 was my error -- it halved a ceiling the model actually uses. OpenCode clamps the wire value at 32000 regardless, verified against the recorder. - options.timeout / chunkTimeout per local provider. Worst observed time-to-first-token on these endpoints is 209s, and OpenCode does not retry a chunk timeout at all (anomalyco/opencode#20466), so a default in the tens of seconds would silently kill healthy turns. - configs/opencode/plugins/local-stall-alert.mjs raises a desktop notification on session.error, with a one-minute cooldown so an outage does not fire forty times. It fixes nothing upstream; it makes a silent failure announce itself. - Dropped the `instructions` entry. ~/.config/opencode/AGENTS.md is a symlink to the same file OpenCode auto-loads, so every house rule was being sent twice: 12,555 duplicated characters in every request, 15% of the system prompt. Verified end to end -- the system prompt fell from 81,046 to 67,421 chars and the whole request from 121,076 to 106,121 bytes. - Disabled local-ai-delegate_run_local_plan: never called once in 47,240 recorded tool calls. Three more never-called tools (list_mcp_resources, list_mcp_resource_templates, read_mcp_resource) CANNOT be disabled from config -- `mcp*`, `*resource*` and `*mcp_resource*` were each tested against the recorder and left all three in place. A test now rejects those dead patterns so the config never claims coverage it does not have. - small_model moved off the paid cloud; 21,734 title generations were going there. - The mlx launchers set --max-tokens (server default is 512, and applies to any request that omits it -- a trap for curl probes and new clients, not for OpenCode which sends its own) and gain optional --chat-template-args. DELIBERATELY unset by default: thinking off measured 89% -> 78% recall on the code-review eval, so it belongs on a bulk lane, never on the reviewer. Takes effect on the next server restart; nothing restarted here. - rotate-logs.sh now covers OpenCode's log directory. Its log was one unrotated 374 MB file, mostly 137,653 "duplicate skill name" warnings. Rotated in place. Not done, deliberately: `steps` stays unset, because capping it forces a text-only summary mid-task, which looks exactly like the stall being fixed. And the duplicate skills themselves are spread across ~/.agents, ~/.claude, ~/.opencode and more -- shared with four other agents, so consolidating them is the operator's call, not a change to make from inside this repo. Co-authored-by: Claude Opus 5 (1M context) <[email protected]>
1 parent 452ee8e commit f4c9b77

10 files changed

Lines changed: 425 additions & 30 deletions

‎configs/opencode/opencode.json.example‎

Lines changed: 75 additions & 20 deletions
Original file line numberDiff line numberDiff line change
@@ -1,71 +1,120 @@
11
{
22
"$schema": "https://opencode.ai/config.json",
3-
"small_model": "glm-meter/glm-5.2",
3+
"small_model": "local-orch//Users/kenallred/ai/models/orchestrator-qwen36-35b-a3b-heretic-mixed9",
44
"compaction": {
55
"auto": true,
66
"prune": true,
77
"reserved": 16000
88
},
99
"agent": {
1010
"review": {
11-
"description": "Lean one-shot code reviewer for MMR — no tools, minimal context, smaller prompts.",
11+
"description": "Lean one-shot code reviewer for MMR \u2014 no tools, minimal context, smaller prompts.",
1212
"prompt": "You are a precise senior code reviewer. Review ONLY the diff/code provided in the message and report concrete, actionable issues. Do not call tools.",
1313
"tools": {
14-
"write": false, "edit": false, "patch": false, "bash": false,
15-
"read": false, "grep": false, "glob": false, "list": false,
16-
"webfetch": false, "task": false, "todowrite": false, "todoread": false
14+
"write": false,
15+
"edit": false,
16+
"patch": false,
17+
"bash": false,
18+
"read": false,
19+
"grep": false,
20+
"glob": false,
21+
"list": false,
22+
"webfetch": false,
23+
"task": false,
24+
"todowrite": false,
25+
"todoread": false
1726
}
1827
}
1928
},
2029
"provider": {
2130
"local-orch": {
2231
"npm": "@ai-sdk/openai-compatible",
2332
"name": "Local Orchestrator (metered)",
24-
"options": { "baseURL": "http://127.0.0.1:9001/v1", "apiKey": "opencode" },
33+
"options": {
34+
"baseURL": "http://127.0.0.1:9001/v1",
35+
"apiKey": "opencode",
36+
"timeout": 1800000,
37+
"chunkTimeout": 300000
38+
},
2539
"models": {
2640
"/Users/kenallred/ai/models/orchestrator-qwen36-35b-a3b-heretic-mixed9": {
27-
"name": "Orchestrator · Qwen3.6-35B-A3B 9-bit (M1)",
28-
"limit": { "context": 131072, "output": 16384 }
41+
"name": "Orchestrator \u00b7 Qwen3.6-35B-A3B 9-bit (M1)",
42+
"limit": {
43+
"context": 131072,
44+
"output": 32000
45+
}
2946
}
3047
}
3148
},
3249
"local-dev": {
3350
"npm": "@ai-sdk/openai-compatible",
3451
"name": "Local Developer (metered)",
35-
"options": { "baseURL": "http://127.0.0.1:9002/v1", "apiKey": "opencode" },
52+
"options": {
53+
"baseURL": "http://127.0.0.1:9002/v1",
54+
"apiKey": "opencode",
55+
"timeout": 1800000,
56+
"chunkTimeout": 300000
57+
},
3658
"models": {
3759
"/Users/admin/ai/models/developer-qwen38-27b-8bit": {
38-
"name": "Developer · Qwen3.8-27B 8-bit (M2)",
39-
"limit": { "context": 131072, "output": 16384 }
60+
"name": "Developer \u00b7 Qwen3.8-27B 8-bit (M2)",
61+
"limit": {
62+
"context": 131072,
63+
"output": 32000
64+
}
4065
}
4166
}
4267
},
4368
"local-review": {
4469
"npm": "@ai-sdk/openai-compatible",
4570
"name": "Local Reviewer (metered, via review router)",
46-
"options": { "baseURL": "http://127.0.0.1:9006/v1", "apiKey": "opencode" },
71+
"options": {
72+
"baseURL": "http://127.0.0.1:9006/v1",
73+
"apiKey": "opencode",
74+
"timeout": 1800000,
75+
"chunkTimeout": 300000
76+
},
4777
"models": {
4878
"/Users/admin/ai/models/developer-qwen38-27b-8bit": {
49-
"name": "Reviewer · Qwen3.8-27B 8-bit (M2)",
50-
"limit": { "context": 131072, "output": 16384 }
79+
"name": "Reviewer \u00b7 Qwen3.8-27B 8-bit (M2)",
80+
"limit": {
81+
"context": 131072,
82+
"output": 32000
83+
}
5184
}
5285
}
5386
},
5487
"glm-meter": {
5588
"npm": "@ai-sdk/openai-compatible",
5689
"name": "Z.ai via meter (tracked)",
57-
"options": { "baseURL": "http://127.0.0.1:9004/v1", "apiKey": "opencode" },
90+
"options": {
91+
"baseURL": "http://127.0.0.1:9004/v1",
92+
"apiKey": "opencode"
93+
},
5894
"models": {
59-
"glm-5.2": { "name": "GLM-5.2 · metered", "limit": { "context": 1000000, "output": 65536 } }
95+
"glm-5.2": {
96+
"name": "GLM-5.2 \u00b7 metered",
97+
"limit": {
98+
"context": 1000000,
99+
"output": 65536
100+
}
101+
}
60102
}
61103
},
62104
"deepseek-meter": {
63105
"npm": "@ai-sdk/openai-compatible",
64106
"name": "DeepSeek (via meter)",
65-
"options": { "baseURL": "http://127.0.0.1:9005/v1", "apiKey": "opencode" },
107+
"options": {
108+
"baseURL": "http://127.0.0.1:9005/v1",
109+
"apiKey": "opencode"
110+
},
66111
"models": {
67-
"deepseek-v4-pro": { "name": "DeepSeek v4 Pro" },
68-
"deepseek-v4-flash": { "name": "DeepSeek v4 Flash" }
112+
"deepseek-v4-pro": {
113+
"name": "DeepSeek v4 Pro"
114+
},
115+
"deepseek-v4-flash": {
116+
"name": "DeepSeek v4 Flash"
117+
}
69118
}
70119
}
71120
},
@@ -95,5 +144,11 @@
95144
}
96145
}
97146
},
98-
"plugin": ["superpowers@git+https://github.com/obra/superpowers.git"]
147+
"plugin": [
148+
"superpowers@git+https://github.com/obra/superpowers.git",
149+
"/Users/kenallred/Developer/server-setup/configs/opencode/plugins/local-stall-alert.mjs"
150+
],
151+
"tools": {
152+
"local-ai-delegate_run_local_plan": false
153+
}
99154
}
Lines changed: 58 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,58 @@
1+
// local-stall-alert — tell the operator when a turn dies instead of letting it end in silence.
2+
//
3+
// Why this exists. When a local mlx endpoint wedges or 502s, OpenCode retries three times, logs
4+
// `AI_RetryError: Failed after 3 attempts`, and then the turn simply ends. Nothing appears in the
5+
// UI. Measured on this stack: on 2026-08-25 between 00:00 and 04:00 every single local turn failed
6+
// — 40 of 40 — and the only symptom the operator saw was the session going quiet, so they typed
7+
// "continue". Two upstream bugs make this worse and are not fixable from here:
8+
// - SSE/chunk timeouts are not classified as retryable, so the turn dies with no retry
9+
// (ml-explore is not involved; this is anomalyco/opencode#20466)
10+
// - a successful response with zero content reads as "task complete" (opencode#31430)
11+
// This plugin does not fix either. It makes them visible, which is the part that was missing.
12+
//
13+
// Wire it up in opencode.json:
14+
// { "plugin": ["/Users/kenallred/Developer/server-setup/configs/opencode/plugins/local-stall-alert.mjs"] }
15+
16+
const NOTIFY_COOLDOWN_MS = 60_000; // one alert a minute — an outage fires this on every retry
17+
18+
let lastNotifiedAt = 0;
19+
20+
/** Pull a short human-readable reason out of whatever shape the error payload arrives in. */
21+
function describe(properties) {
22+
const error = properties?.error ?? properties;
23+
if (!error) return "unknown error";
24+
const raw =
25+
error.data?.message ??
26+
error.message ??
27+
error.name ??
28+
(typeof error === "string" ? error : JSON.stringify(error));
29+
return String(raw).replace(/\s+/g, " ").slice(0, 160);
30+
}
31+
32+
export const LocalStallAlert = async ({ $ }) => {
33+
return {
34+
event: async ({ event }) => {
35+
if (event?.type !== "session.error") return;
36+
37+
const now = Date.now();
38+
if (now - lastNotifiedAt < NOTIFY_COOLDOWN_MS) return;
39+
lastNotifiedAt = now;
40+
41+
const reason = describe(event.properties);
42+
// `launchpad notify` is the house notifier and is already on PATH for interactive shells.
43+
// Never let a failed notification take down the session that is already having a bad time.
44+
try {
45+
await $`launchpad notify ${`OpenCode turn failed — ${reason}`}`.quiet();
46+
} catch {
47+
try {
48+
await $`osascript -e ${`display notification ${JSON.stringify(reason)} with title "OpenCode turn failed"`}`.quiet();
49+
} catch {
50+
// Both notifiers unavailable (headless run, no launchpad). Nothing further to try;
51+
// the error is still in OpenCode's own log.
52+
}
53+
}
54+
},
55+
};
56+
};
57+
58+
export default LocalStallAlert;

‎docs/dev-stack-models.md‎

Lines changed: 64 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -33,6 +33,70 @@ secret. Providers:
3333
> pins the right model per port (`METER_ORCH_MODEL`, `METER_DEV_MODEL`, `METER_REVIEW_MODEL`), which
3434
> masks a wrong id rather than fixing it — keep these accurate anyway.
3535
36+
### Why local turns used to stop mid-task
37+
38+
The symptom was OpenCode going quiet part-way through a job, needing a typed "continue". Measured
39+
2026-08-25 from OpenCode's own session database (58k assistant turns), its log, and the meter.
40+
41+
**It was almost always the endpoint dying, not OpenCode losing its place.** Turns that never
42+
completed, over 7 days: glm-meter 0.9%, local-orch 0.0%, local-dev 38.1%. Broken down by hour,
43+
2026-08-25 00:00–04:00 shows 40 local turns and 40 failures — exactly the M2 wedge window. Earlier,
44+
5,167 of 5,430 `local-review` stream errors were `Bad Gateway` clustered on Aug 2–6, the M2 outage
45+
that went unreported for four days.
46+
47+
What made it *look* like an OpenCode fault is that it fails silently: three retries, an
48+
`AI_RetryError: Failed after 3 attempts` line in its log, then the turn simply ends with nothing on
49+
screen. `configs/opencode/plugins/local-stall-alert.mjs` now raises a desktop notification on
50+
`session.error` so an outage announces itself.
51+
52+
The second cause was real but rarer: the orchestrator hit its output ceiling and was cut off
53+
mid-work. Only 2 of 34 recorded "continue" nudges followed a local turn, and both had
54+
`finish=length`. `local-orch` truncates on 1.2% of turns against glm-5.2's 0.01%.
55+
56+
Two upstream OpenCode bugs make both worse and **cannot be fixed from here**: SSE/chunk timeouts are
57+
not classified as retryable, so such a turn dies with no retry
58+
([#20466](https://github.com/anomalyco/opencode/issues/20466)); and a successful response carrying
59+
zero content reads as "task complete" ([#31430](https://github.com/anomalyco/opencode/issues/31430)).
60+
The second does not currently affect this stack — zero of 4,315 captured local responses were empty.
61+
62+
### What each local setting is for
63+
64+
| Setting | Value | Why |
65+
|---|---|---|
66+
| `limit.context` | 131,072 | Below the measured crossover where the orchestrator stops being the faster endpoint |
67+
| `limit.output` | 32,000 | The ceiling the orchestrator has actually hit; OpenCode clamps the wire value at 32,000 regardless of a higher number here |
68+
| `options.timeout` | 1,800,000 | 32,000 tokens at ~12 tok/s is ~45 min worst case; 30 min covers real work while still bounding a hang |
69+
| `options.chunkTimeout` | 300,000 | Gap *between* chunks. Worst observed time-to-first-token on these endpoints is 209s |
70+
| `compaction.reserved` | 16,000 | A thinking model needs room to write the summary |
71+
| `tools` | 1 disabled | `local-ai-delegate_run_local_plan` was never called once in 47,240 recorded tool calls, yet shipped in every request |
72+
| `small_model` | local-orch | 21,734 title generations were going to a paid cloud model |
73+
74+
Not set: `steps`. Leaving it unset lets an agent iterate until the model stops, which is what you
75+
want for long local jobs — capping it forces a text-only summary mid-task, which looks exactly like
76+
the stall this section is about.
77+
78+
### Keep the system prompt small
79+
80+
Every local turn pays for the whole request up front. Measured with a throwaway recording endpoint,
81+
the request for the message "say hi" was 121 KB — an 81,046-character system prompt plus 19 tool
82+
definitions, about 29k tokens, roughly 20 seconds of prompt processing before the first token.
83+
84+
15% of that system prompt was the same text twice. `~/.config/opencode/AGENTS.md` is a symlink to
85+
`new-mac/config/agents/AGENTS.md`, OpenCode auto-loads it, and `instructions` named the same file
86+
again by its real path — so every house rule was sent twice. Removing the `instructions` entry cut
87+
12,555 characters per request with nothing lost.
88+
89+
If you add MCP servers, check what they cost here before keeping them: tool definitions alone were
90+
31 KB of that request.
91+
92+
Four tools were never called once in 47,240 recorded tool calls. Only one of them can be turned
93+
off. `local-ai-delegate_run_local_plan` takes the documented `tools` entry because it carries its
94+
MCP server's prefix. The other three — `list_mcp_resources`, `list_mcp_resource_templates` and
95+
`read_mcp_resource` — are OpenCode built-ins that appear whenever any MCP server is configured, and
96+
**they cannot be disabled from config**: `mcp*`, `*resource*` and `*mcp_resource*` were each tested
97+
against a recording endpoint and left all three in the tool list. The only way to drop them is to
98+
remove every MCP server, which costs more than it saves.
99+
36100
### Why the local models carry a context cap
37101

38102
The mlx servers leak Metal buffer descriptors while decoding and eventually die with

‎scripts/rotate-logs.sh‎

Lines changed: 21 additions & 9 deletions
Original file line numberDiff line numberDiff line change
@@ -9,18 +9,30 @@ set -uo pipefail
99
# place keeps their file descriptors valid.
1010
#
1111
# Keeps exactly one previous copy per log (<name>.1), overwritten each rotation.
12-
# Env overrides: LOG_DIR, MAX_BYTES.
12+
# Env overrides: LOG_DIR, EXTRA_LOG_DIRS (colon-separated), MAX_BYTES.
1313

1414
LOG_DIR="${LOG_DIR:-$HOME/ai/logs}"
15+
# OpenCode writes one unrotated file and never trims it: found at 374 MB on 2026-08-25, most of it
16+
# 137,653 "duplicate skill name" warnings emitted on every start. Same copy-truncate reasoning as
17+
# above — the running process holds the file open.
18+
EXTRA_LOG_DIRS="${EXTRA_LOG_DIRS:-$HOME/.local/share/opencode/log}"
1519
MAX_BYTES="${MAX_BYTES:-52428800}" # 50 MB
1620

17-
[ -d "$LOG_DIR" ] || exit 0 # nothing logged yet — not an error
21+
rotate_dir() {
22+
local dir="$1" log size
23+
[ -d "$dir" ] || return 0 # nothing logged yet — not an error
24+
for log in "$dir"/*.log; do
25+
[ -f "$log" ] || continue # no matches: the glob stays literal
26+
size=$(stat -f%z "$log" 2>/dev/null || echo 0)
27+
[ "$size" -gt "$MAX_BYTES" ] || continue
28+
cp "$log" "$log.1" && : > "$log" &&
29+
printf '%s rotated %s (%s bytes)\n' "$(date '+%Y-%m-%dT%H:%M:%S%z')" "$(basename "$log")" "$size"
30+
done
31+
}
1832

19-
for log in "$LOG_DIR"/*.log; do
20-
[ -f "$log" ] || continue # no matches: the glob stays literal
21-
size=$(stat -f%z "$log" 2>/dev/null || echo 0)
22-
[ "$size" -gt "$MAX_BYTES" ] || continue
23-
cp "$log" "$log.1" && : > "$log" &&
24-
printf '%s rotated %s (%s bytes)\n' "$(date '+%Y-%m-%dT%H:%M:%S%z')" "$(basename "$log")" "$size"
25-
done
33+
rotate_dir "$LOG_DIR"
34+
# Split on ':' without leaking IFS into anything that runs after this loop.
35+
while IFS= read -r extra; do
36+
[ -n "$extra" ] && rotate_dir "$extra"
37+
done <<< "$(printf '%s' "$EXTRA_LOG_DIRS" | tr ':' '\n')"
2638
exit 0

‎scripts/start-developer.sh‎

Lines changed: 15 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -24,6 +24,19 @@ if [ -n "${DRAFT_MODEL:-}" ]; then
2424
DRAFT_ARGS=(--draft-model "$DRAFT_MODEL" --num-draft-tokens "${NUM_DRAFT_TOKENS:-3}")
2525
fi
2626

27+
# mlx_lm.server's own --max-tokens default is 512, and it applies to any request that does not
28+
# send max_tokens of its own. OpenCode and the Phase 8 enrichment script both send one, so this is
29+
# a floor for everything else (curl probes, smoke tests, new clients) rather than a cap on them.
30+
MAX_TOKENS="${MLX_MAX_TOKENS:-8192}"
31+
# Chat-template arguments, passed to the model's Jinja template as JSON — this is how to set
32+
# something like {"enable_thinking": false} for a whole endpoint instead of per request.
33+
# UNSET BY DEFAULT ON PURPOSE: turning thinking off measured 89% -> 78% recall on the code-review
34+
# eval, so it belongs on a bulk-classification lane, never on the reviewer or developer.
35+
TEMPLATE_ARGS=()
36+
if [ -n "${MLX_CHAT_TEMPLATE_ARGS:-}" ]; then
37+
TEMPLATE_ARGS=(--chat-template-args "$MLX_CHAT_TEMPLATE_ARGS")
38+
fi
39+
2740
# Abort only if OUR mlx_lm server is already answering here (avoid a duplicate). A bare lsof
2841
# check false-positives on unrelated listeners sharing the port (e.g. a Docker container on
2942
# 0.0.0.0:PORT), which would crash-loop under launchd. Exit 0 (not 1) so "already running" isn't
@@ -39,6 +52,8 @@ exec mlx_lm.server \
3952
--model "$MODEL" \
4053
--host "$HOST" \
4154
--port "$PORT" \
55+
--max-tokens "$MAX_TOKENS" \
56+
${TEMPLATE_ARGS[@]+"${TEMPLATE_ARGS[@]}"} \
4257
--prompt-cache-size "$PROMPT_CACHE_SIZE" \
4358
--prompt-cache-bytes "$PROMPT_CACHE_BYTES" \
4459
${DRAFT_ARGS[@]+"${DRAFT_ARGS[@]}"} \

‎scripts/start-orchestrator.sh‎

Lines changed: 15 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -13,6 +13,19 @@ HOST="${ORCH_HOST:-127.0.0.1}"
1313
PROMPT_CACHE_SIZE="${PROMPT_CACHE_SIZE:-10}"
1414
PROMPT_CACHE_BYTES="${PROMPT_CACHE_BYTES:-8GB}"
1515

16+
# mlx_lm.server's own --max-tokens default is 512, and it applies to any request that does not
17+
# send max_tokens of its own. OpenCode and the Phase 8 enrichment script both send one, so this is
18+
# a floor for everything else (curl probes, smoke tests, new clients) rather than a cap on them.
19+
MAX_TOKENS="${MLX_MAX_TOKENS:-8192}"
20+
# Chat-template arguments, passed to the model's Jinja template as JSON — this is how to set
21+
# something like {"enable_thinking": false} for a whole endpoint instead of per request.
22+
# UNSET BY DEFAULT ON PURPOSE: turning thinking off measured 89% -> 78% recall on the code-review
23+
# eval, so it belongs on a bulk-classification lane, never on the reviewer or developer.
24+
TEMPLATE_ARGS=()
25+
if [ -n "${MLX_CHAT_TEMPLATE_ARGS:-}" ]; then
26+
TEMPLATE_ARGS=(--chat-template-args "$MLX_CHAT_TEMPLATE_ARGS")
27+
fi
28+
1629
# Abort only if OUR OpenAI-compatible server is already answering here (avoid a duplicate).
1730
# A bare lsof check false-positives on unrelated listeners that may share the port on another
1831
# interface (e.g. a Docker/OrbStack container publishing 0.0.0.0:PORT), which would crash-loop
@@ -32,6 +45,8 @@ exec mlx_lm.server \
3245
--model "$MODEL" \
3346
--host "$HOST" \
3447
--port "$PORT" \
48+
--max-tokens "$MAX_TOKENS" \
49+
${TEMPLATE_ARGS[@]+"${TEMPLATE_ARGS[@]}"} \
3550
--prompt-cache-size "$PROMPT_CACHE_SIZE" \
3651
--prompt-cache-bytes "$PROMPT_CACHE_BYTES" \
3752
>> "$HOME/ai/logs/orchestrator.log" 2>&1

0 commit comments

Comments
 (0)