Skip to content

Commit 38b3e73

Browse files
steipetesteipete
authored
fix: improve gateway protocol mismatch diagnostics (#82908)
* fix: improve gateway protocol mismatch diagnostics * test: cover daemon deep connection diagnostics * fix: normalize mapped loopback gateway clients
1 parent 926a5a8 commit 38b3e73

14 files changed

Lines changed: 801 additions & 6 deletions

‎CHANGELOG.md‎

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -19,6 +19,7 @@ Docs: https://docs.openclaw.ai
1919
- CLI/media: accept HTTP(S) URLs in `openclaw infer image describe --file`, fetching remote images through the guarded media path instead of treating URLs as local files. Fixes #82837. (#82854) Thanks @neeravmakwana.
2020
- Agents/subagents: keep session-backed parent runs active when the child wait call times out before the child session has actually settled, so late subagent completions are reconciled instead of being lost. Fixes #82787. Thanks @ramitrkar-hash.
2121
- Control UI: advertise shared Gateway protocol constants in browser connect frames, fixing protocol mismatch handshakes after the protocol 5 bump. Fixes #82882. Thanks @galiniliev.
22+
- Gateway: add rollback protocol-mismatch diagnostics, including client protocol ranges in Gateway logs and deep status/doctor hints for stale client processes. Fixes #82841. (#82908)
2223
- Agents/subagents: route group/channel subagent completions through message-tool-only handoffs when required and keep active-requester wake failures from dropping completion delivery. Fixes #82803. Thanks @galiniliev, @yozakura-ava, and @moeedahmed.
2324
- Memory-core: scan persisted memory source sessions on startup, comparing on-disk transcripts against the index and marking only missing/newer/resized files dirty for incremental sync. Fixes #82341. (#82341) Thanks @giodl73-repo.
2425
- Telegram: keep the top-level default account in the account list when named accounts or bindings are added alongside top-level credentials, preserving default polling while still letting named-only configs resolve to a single account. Fixes #82794. (#82794) Thanks @giodl73-repo.

‎docs/gateway/troubleshooting.md‎

Lines changed: 26 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -86,6 +86,32 @@ openclaw config get meta.lastTouchedVersion
8686
For intentional downgrade or emergency recovery only, set `OPENCLAW_ALLOW_OLDER_BINARY_DESTRUCTIVE_ACTIONS=1` for the single command. Leave it unset for normal operation.
8787
</Warning>
8888

89+
## Protocol mismatch after rollback
90+
91+
Use this when logs keep printing `protocol mismatch` after you downgrade or roll back OpenClaw. This means an older Gateway is running, but a newer local client process is still trying to reconnect with a protocol range that the older Gateway cannot speak.
92+
93+
```bash
94+
openclaw --version
95+
which -a openclaw
96+
openclaw gateway status --deep
97+
openclaw doctor --deep
98+
openclaw logs --follow
99+
```
100+
101+
Look for:
102+
103+
- `protocol mismatch ... client=... v<version> min=<n> max=<n> expected=<n>` in Gateway logs.
104+
- `Established clients:` in `openclaw gateway status --deep` or `Gateway clients` in `openclaw doctor --deep`. This lists active TCP clients connected to the Gateway port, including PIDs and command lines when the OS allows it.
105+
- A client process whose command line points at the newer OpenClaw install or wrapper you rolled back from.
106+
107+
Fix:
108+
109+
1. Stop or restart the stale OpenClaw client process shown by `gateway status --deep`.
110+
2. Restart apps or wrappers that embed OpenClaw, such as local dashboards, editors, app-server helpers, or long-running `openclaw logs --follow` shells.
111+
3. Re-run `openclaw gateway status --deep` or `openclaw doctor --deep` and confirm the stale client PID is gone.
112+
113+
Do not make an older Gateway accept a newer incompatible protocol. Protocol bumps protect the wire contract; rollback recovery is a process/version cleanup problem.
114+
89115
## Skill symlink skipped as path escape
90116

91117
Use this when logs include:

‎src/cli/daemon-cli.coverage.test.ts‎

Lines changed: 6 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -28,6 +28,10 @@ const inspectPortUsage = vi.fn(async (port: number) => ({
2828
listeners: [],
2929
hints: [],
3030
}));
31+
const inspectPortConnections = vi.fn(async (port: number) => ({
32+
port,
33+
connections: [],
34+
}));
3135

3236
function collectMatching<T, U>(
3337
items: readonly T[],
@@ -114,6 +118,7 @@ vi.mock("../daemon/inspect.js", () => ({
114118
}));
115119

116120
vi.mock("../infra/ports.js", () => ({
121+
inspectPortConnections: (port: number) => inspectPortConnections(port),
117122
inspectPortUsage: (port: number) => inspectPortUsage(port),
118123
formatPortDiagnostics: () => ["Port 18789 is already in use."],
119124
}));
@@ -192,6 +197,7 @@ describe("daemon-cli coverage", () => {
192197
serviceReadCommand.mockResolvedValue(null);
193198
resolveGatewayProbeAuthSafeWithSecretInputs.mockClear();
194199
findExtraGatewayServices.mockClear();
200+
inspectPortConnections.mockClear();
195201
buildGatewayInstallPlan.mockClear();
196202
});
197203

‎src/cli/daemon-cli/status.gather.test.ts‎

Lines changed: 62 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -4,6 +4,7 @@ import path from "node:path";
44
import { afterEach, beforeEach, describe, expect, it, vi } from "vitest";
55
import type { StaleOpenClawUpdateLaunchdJob } from "../../daemon/launchd.js";
66
import { createMockGatewayService } from "../../daemon/service.test-helpers.js";
7+
import type { PortConnections } from "../../infra/ports.js";
78
import type { GatewayRestartHandoff } from "../../infra/restart-handoff.js";
89
import { captureEnv } from "../../test-utils/env.js";
910
import { VERSION } from "../../version.js";
@@ -38,6 +39,12 @@ const inspectPortUsage = vi.fn(async (port: number) => ({
3839
listeners: [],
3940
hints: [],
4041
}));
42+
const inspectPortConnections = vi.fn<(port: number) => Promise<PortConnections>>(
43+
async (port: number) => ({
44+
port,
45+
connections: [],
46+
}),
47+
);
4148
const readLastGatewayErrorLine = vi.fn(async (_env?: NodeJS.ProcessEnv) => null);
4249
const readGatewayRestartHandoffSync = vi.fn<
4350
(_env?: NodeJS.ProcessEnv) => GatewayRestartHandoff | null
@@ -166,6 +173,7 @@ vi.mock("../../gateway/net.js", () => ({
166173
}));
167174

168175
vi.mock("../../infra/ports.js", () => ({
176+
inspectPortConnections: (port: number) => inspectPortConnections(port),
169177
inspectPortUsage: (port: number) => inspectPortUsage(port),
170178
formatPortDiagnostics: () => [],
171179
}));
@@ -222,6 +230,7 @@ describe("gatherDaemonStatus", () => {
222230
findStaleOpenClawUpdateLaunchdJobs.mockResolvedValue([]);
223231
loadGatewayTlsRuntime.mockClear();
224232
inspectGatewayRestart.mockClear();
233+
inspectPortConnections.mockClear();
225234
readGatewayRestartHandoffSync.mockClear();
226235
readConfigFileSnapshotCalls.mockClear();
227236
loadConfigCalls.mockClear();
@@ -484,6 +493,59 @@ describe("gatherDaemonStatus", () => {
484493

485494
expect(readGatewayRestartHandoffSync).not.toHaveBeenCalled();
486495
expect(findStaleOpenClawUpdateLaunchdJobs).not.toHaveBeenCalled();
496+
expect(inspectPortConnections).not.toHaveBeenCalled();
497+
});
498+
499+
it("surfaces established gateway connections during deep status", async () => {
500+
inspectPortConnections.mockResolvedValueOnce({
501+
port: 19001,
502+
connections: [
503+
{
504+
pid: 4242,
505+
ppid: 1,
506+
command: "node",
507+
commandLine: "node /tmp/newer-openclaw/dist/index.js logs --follow",
508+
address: "TCP 127.0.0.1:50123->127.0.0.1:19001 (ESTABLISHED)",
509+
direction: "client",
510+
},
511+
],
512+
});
513+
514+
const status = await gatherDaemonStatus({
515+
rpc: {},
516+
probe: false,
517+
deep: true,
518+
});
519+
520+
expect(inspectPortConnections).toHaveBeenCalledWith(19001);
521+
expect(status.connections?.established).toEqual([
522+
{
523+
pid: 4242,
524+
ppid: 1,
525+
command: "node",
526+
commandLine: "node /tmp/newer-openclaw/dist/index.js logs --follow",
527+
address: "TCP 127.0.0.1:50123->127.0.0.1:19001 (ESTABLISHED)",
528+
direction: "client",
529+
},
530+
]);
531+
});
532+
533+
it("skips established gateway connection scans for remote gateway status", async () => {
534+
daemonLoadedConfig = {
535+
gateway: {
536+
mode: "remote",
537+
remote: { url: "wss://gateway.example" },
538+
},
539+
};
540+
541+
const status = await gatherDaemonStatus({
542+
rpc: {},
543+
probe: false,
544+
deep: true,
545+
});
546+
547+
expect(inspectPortConnections).not.toHaveBeenCalled();
548+
expect(status.connections).toBeUndefined();
487549
});
488550

489551
it("uses the fast config path for plain same-file status reads", async () => {

‎src/cli/daemon-cli/status.gather.ts‎

Lines changed: 33 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -26,7 +26,9 @@ import {
2626
import { parseStrictPositiveInteger } from "../../infra/parse-finite-number.js";
2727
import {
2828
formatPortDiagnostics,
29+
inspectPortConnections,
2930
inspectPortUsage,
31+
type PortConnection,
3032
type PortListener,
3133
type PortUsageStatus,
3234
} from "../../infra/ports.js";
@@ -294,6 +296,10 @@ export type DaemonStatus = {
294296
listeners: PortListener[];
295297
hints: string[];
296298
};
299+
connections?: {
300+
port: number;
301+
established: PortConnection[];
302+
};
297303
lastError?: string;
298304
rpc?: {
299305
ok: boolean;
@@ -460,6 +466,27 @@ async function inspectDaemonPortStatuses(params: {
460466
};
461467
}
462468

469+
async function inspectEstablishedGatewayClients(params: {
470+
daemonPort: number;
471+
deep?: boolean;
472+
gatewayMode?: string;
473+
}): Promise<DaemonStatus["connections"] | undefined> {
474+
if (params.deep !== true || params.gatewayMode === "remote") {
475+
return undefined;
476+
}
477+
const result = await inspectPortConnections(params.daemonPort).catch(() => null);
478+
const establishedClients = result?.connections.filter(
479+
(connection) => connection.direction !== "server",
480+
);
481+
if (!result || !establishedClients || establishedClients.length === 0) {
482+
return undefined;
483+
}
484+
return {
485+
port: result.port,
486+
established: establishedClients,
487+
};
488+
}
489+
463490
export async function gatherDaemonStatus(
464491
opts: {
465492
rpc: GatewayRpcOpts;
@@ -508,6 +535,11 @@ export async function gatherDaemonStatus(
508535
daemonPort,
509536
cliPort,
510537
});
538+
const establishedClients = await inspectEstablishedGatewayClients({
539+
daemonPort,
540+
deep: opts.deep,
541+
gatewayMode: daemonCfg.gateway?.mode,
542+
});
511543

512544
const extraServices = opts.deep
513545
? await loadDaemonInspectModule()
@@ -618,6 +650,7 @@ export async function gatherDaemonStatus(
618650
gateway,
619651
port: portStatus,
620652
...(portCliStatus ? { portCli: portCliStatus } : {}),
653+
...(establishedClients ? { connections: establishedClients } : {}),
621654
lastError,
622655
...(rpc
623656
? {

‎src/cli/daemon-cli/status.print.test.ts‎

Lines changed: 42 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -131,6 +131,48 @@ describe("printDaemonStatus", () => {
131131
expectMockLineContains(runtime.error, formatCliCommand("openclaw gateway restart"));
132132
});
133133

134+
it("prints established gateway client guidance gathered by deep status", () => {
135+
printDaemonStatus(
136+
{
137+
service: {
138+
label: "LaunchAgent",
139+
loaded: true,
140+
loadedText: "loaded",
141+
notLoadedText: "not loaded",
142+
runtime: { status: "running", pid: 8000 },
143+
},
144+
gateway: {
145+
bindMode: "loopback",
146+
bindHost: "127.0.0.1",
147+
port: 18789,
148+
portSource: "env/config",
149+
probeUrl: "ws://127.0.0.1:18789",
150+
},
151+
connections: {
152+
port: 18789,
153+
established: [
154+
{
155+
pid: 4242,
156+
ppid: 1,
157+
command: "node",
158+
commandLine: "/tmp/newer-openclaw/bin/openclaw logs --follow",
159+
address: "TCP 127.0.0.1:50123->127.0.0.1:18789 (ESTABLISHED)",
160+
direction: "client",
161+
},
162+
],
163+
},
164+
extraServices: [],
165+
},
166+
{ json: false },
167+
);
168+
169+
expectMockLineContains(runtime.log, "Established clients: 1");
170+
expectMockLineContains(runtime.log, "pid=4242");
171+
expectMockLineContains(runtime.log, "newer-openclaw");
172+
expectMockLineContains(runtime.log, "client");
173+
expectMockLineContains(runtime.log, "protocol mismatch after rollback");
174+
});
175+
134176
it("prints stale updater launchd job guidance", () => {
135177
printDaemonStatus(
136178
{

‎src/cli/daemon-cli/status.print.ts‎

Lines changed: 34 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -72,6 +72,20 @@ function formatCliVersionLine(cli: DaemonStatus["cli"]): string | null {
7272
return cli.entrypoint ? `${cli.version} (${shortenHomePath(cli.entrypoint)})` : cli.version;
7373
}
7474

75+
function formatConnectionLine(
76+
connection: NonNullable<DaemonStatus["connections"]>["established"][number],
77+
) {
78+
const pid = connection.pid ? `pid=${connection.pid}` : "pid=?";
79+
const ppid = connection.ppid ? ` ppid=${connection.ppid}` : "";
80+
const direction = ` ${connection.direction}`;
81+
const command = connection.command ? ` ${connection.command}` : "";
82+
const address = connection.address ? ` ${connection.address}` : "";
83+
const commandLine = connection.commandLine
84+
? ` cmd=${shortenHomePath(connection.commandLine)}`
85+
: "";
86+
return `${pid}${ppid}${direction}${command}${address}${commandLine}`;
87+
}
88+
7589
export function printDaemonStatus(status: DaemonStatus, opts: { json: boolean }) {
7690
if (opts.json) {
7791
const sanitized = sanitizeDaemonStatusForJson(status);
@@ -285,6 +299,26 @@ export function printDaemonStatus(status: DaemonStatus, opts: { json: boolean })
285299
spacer();
286300
}
287301

302+
if (status.connections?.established.length) {
303+
defaultRuntime.log(
304+
`${label("Established clients:")} ${infoText(String(status.connections.established.length))}`,
305+
);
306+
for (const connection of status.connections.established.slice(0, 8)) {
307+
defaultRuntime.log(` ${infoText(formatConnectionLine(connection))}`);
308+
}
309+
if (status.connections.established.length > 8) {
310+
defaultRuntime.log(
311+
` ${infoText(`... ${status.connections.established.length - 8} more connection(s)`)}`,
312+
);
313+
}
314+
defaultRuntime.log(
315+
warnText(
316+
"If logs show protocol mismatch after rollback, stop stale OpenClaw client processes listed here and re-run gateway status.",
317+
),
318+
);
319+
spacer();
320+
}
321+
288322
const systemdUnavailable =
289323
process.platform === "linux" && isSystemdUnavailableDetail(service.runtime?.detail);
290324
if (systemdUnavailable) {

0 commit comments

Comments
 (0)