Skip to content

Commit cfad2c4

Browse files
committed
fix(server): preview resize and status report the real size of desktop tabs
In the desktop app the server drives each Browser panel tab through CDP, and the desktop draws it in a webview. Playwright has no viewport for those pages, so preview_resize always answered with the 1280x800 fallback. preview_status read innerWidth/innerHeight in a new isolated world on every call and silently kept the last cached size when the read failed or ran before the desktop had drawn, so after a freeform 1280x760 resize that the page had already applied, status could still report the old size. Status, resize, and the reads behind them now use what the server already knows: - Status keeps the measured size and reads the page again only after a page resize, load, or dialog close marks it stale. This holds for fixed settings (freeform and preset) too, because a page can keep another size than the one asked for: a tab the desktop does not draw kept 560x760 after an 844x390 request (#22942). A user's change to the setting is recorded as soon as its session event arrives, so status does not wait behind a running agent action to see it. A read that fails or measures 0x0 leaves it stale, so the next status retries. A status that arrives while a read is running waits for that read instead of answering the size from before it. - Resize captures a signal that the desktop page's next Page.frameResized settles, before it publishes the new setting. When the CSS size changes it waits for that resize, so the agent's next snapshot sees the new size, the same as Playwright tabs where setViewportSize is awaited. The wait is bounded by min(1 s, half the time left before the request's deadline). Every resize then reads the page, even if no resize arrived, within the rest of that time, so a resize the page never draws still reports the size the page has and answers before the request's deadline, even when it waited in the tab's control queue. The stale flag rides on the existing CDP listeners rather than starting a read on every page resize, so a user dragging the panel costs nothing until the next status. Remote viewers of desktop tabs keep their eager size pushes unchanged. Playwright-owned tabs keep their current paths. Status on a desktop tab now makes no CDP calls unless the page has changed since the last read. User story: When my agent resizes a preview in the desktop app's Browser panel, it is told the size the page really has, so it no longer makes layout decisions from a wrong 1280x800 or stale size.
1 parent 09a54f0 commit cfad2c4

2 files changed

Lines changed: 145 additions & 6 deletions

File tree

‎apps/server/src/preview/ServerBrowser.test.ts‎

Lines changed: 88 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -13,6 +13,7 @@ import {
1313
PreviewTabId,
1414
ProviderInstanceId,
1515
ThreadId,
16+
type PreviewAutomationResizeResult,
1617
type PreviewAutomationSnapshot,
1718
type PreviewAutomationStatus,
1819
} from "@t3tools/contracts";
@@ -1892,6 +1893,93 @@ it.effect(
18921893
).pipe(Effect.provide(layer)),
18931894
);
18941895

1896+
it.effect("a desktop tab reports the size the desktop draws after a resize", () =>
1897+
Effect.scoped(
1898+
Effect.gen(function* () {
1899+
const { broker } = yield* ready;
1900+
const manager = yield* Manager.PreviewManager;
1901+
desktopRendersNext = true;
1902+
const opened = yield* broker.invoke<PreviewAutomationStatus>({
1903+
scope,
1904+
operation: "open",
1905+
input: { reuseExistingTab: false, show: false },
1906+
});
1907+
const tabId = PreviewTabId.make(opened.tabId!);
1908+
const { context } = desktopConnections.at(-1)!;
1909+
const cdp = context.sessions[0]!;
1910+
const events = yield* manager.subscribeEvents;
1911+
const status = broker
1912+
.invoke<PreviewAutomationStatus>({ scope, tabId, operation: "status", input: {} })
1913+
.pipe(Effect.map((result) => result.viewport));
1914+
// The desktop draws the new size once the session update reaches it.
1915+
const resize = (input: Record<string, unknown>, width: number, height: number) =>
1916+
Effect.gen(function* () {
1917+
const result = yield* broker
1918+
.invoke<PreviewAutomationResizeResult>({ scope, tabId, operation: "resize", input })
1919+
.pipe(Effect.forkScoped);
1920+
let event = yield* PubSub.take(events);
1921+
while (event.type !== "resized" || event.tabId !== tabId) {
1922+
event = yield* PubSub.take(events);
1923+
}
1924+
Object.assign(context.cssViewport, { width, height });
1925+
yield* Effect.promise(() => cdp.emitAsync("Page.frameResized"));
1926+
return (yield* Fiber.join(result)).viewport;
1927+
});
1928+
1929+
expect(yield* resize({ mode: "freeform", width: 1280, height: 760 }, 1280, 760)).toEqual({
1930+
width: 1280,
1931+
height: 760,
1932+
});
1933+
desktopSizeAnswer = () => Promise.reject(new Error("Cannot find context with specified id"));
1934+
expect(yield* status).toEqual({ width: 1280, height: 760 });
1935+
desktopSizeAnswer = null;
1936+
// A page can keep another size than the fixed one asked for; both report the page's.
1937+
expect(yield* resize({ mode: "freeform", width: 844, height: 390 }, 560, 760)).toEqual({
1938+
width: 560,
1939+
height: 760,
1940+
});
1941+
expect(yield* status).toEqual({ width: 560, height: 760 });
1942+
expect(yield* resize({ mode: "fill" }, 1234, 700)).toEqual({ width: 1234, height: 700 });
1943+
// No page resize or load since that read, so status answers without reading again.
1944+
let reads = 0;
1945+
desktopSizeAnswer = async () => {
1946+
reads += 1;
1947+
return { result: { value: { width: 1, height: 1 } } };
1948+
};
1949+
expect(yield* status).toEqual({ width: 1234, height: 700 });
1950+
expect(reads).toBe(0);
1951+
// A status during a read waits for it rather than answer the size before it.
1952+
const pending = yield* Queue.unbounded<PromiseWithResolvers<Record<string, unknown>>>();
1953+
desktopSizeAnswer = () => {
1954+
const read = Promise.withResolvers<Record<string, unknown>>();
1955+
Queue.offerUnsafe(pending, read);
1956+
return read.promise;
1957+
};
1958+
const refill = yield* broker
1959+
.invoke<PreviewAutomationResizeResult>({
1960+
scope,
1961+
tabId,
1962+
operation: "resize",
1963+
input: { mode: "fill" },
1964+
})
1965+
.pipe(Effect.forkScoped);
1966+
const read = yield* Queue.take(pending);
1967+
// Status reads the URL just before it picks the size.
1968+
const statusStarted = Promise.withResolvers<void>();
1969+
const url = context.page.url();
1970+
using _url = vi.spyOn(context.page, "url").mockImplementationOnce(() => {
1971+
statusStarted.resolve();
1972+
return url;
1973+
});
1974+
const during = yield* Effect.forkScoped(status);
1975+
yield* Effect.promise(() => statusStarted.promise);
1976+
read.resolve({ result: { value: { width: 1100, height: 650 } } });
1977+
expect(yield* Fiber.join(during)).toEqual({ width: 1100, height: 650 });
1978+
expect((yield* Fiber.join(refill)).viewport).toEqual({ width: 1100, height: 650 });
1979+
}),
1980+
).pipe(Effect.provide(layer)),
1981+
);
1982+
18951983
it.effect("a stalled desktop viewer still gets its size, control and reconnect", () =>
18961984
Effect.scoped(
18971985
Effect.gen(function* () {

‎apps/server/src/preview/ServerBrowser.ts‎

Lines changed: 57 additions & 6 deletions
Original file line numberDiff line numberDiff line change
@@ -105,6 +105,8 @@ const AGENT_CURSOR_MOVE_MS = 160;
105105
const AGENT_CURSOR_CLICK_LEAD_MS = 40;
106106
/** A dialog blocks page script, so a desktop size read gives up after this. */
107107
const DESKTOP_VIEWPORT_READ_TIMEOUT = "1 second";
108+
/** A desktop page that keeps its size draws nothing on a resize, so the wait gives up. */
109+
const DESKTOP_RESIZE_WAIT_MS = 1000;
108110

109111
const sleepUntil = (deadline: number) =>
110112
new Promise<void>((resolve) => setTimeout(resolve, Math.max(0, deadline - Date.now())));
@@ -328,6 +330,10 @@ interface ServerTab {
328330
desktopViewportRead: Promise<ServerTab["desktopViewport"]>;
329331
/** A read queued behind `desktopViewportRead`, shared by every caller that waits for it. */
330332
desktopViewportNext: Promise<ServerTab["desktopViewport"]> | null;
333+
/** A page resize or load since the last read started; status measures again. */
334+
desktopViewportStale: boolean;
335+
/** Settles at the desktop page's next main-frame resize. */
336+
desktopResize: PromiseWithResolvers<void>;
331337
loading: boolean;
332338
closing: boolean;
333339
recording: Recording | null;
@@ -666,13 +672,15 @@ const make = Effect.gen(function* () {
666672
const height = size?.height;
667673
if (typeof width === "number" && width > 0 && typeof height === "number" && height > 0)
668674
tab.desktopViewport = { width, height };
675+
else tab.desktopViewportStale = true;
669676
return tab.desktopViewport;
670677
};
671678

672679
// Reads run one at a time, and callers that arrive during a read share the next one.
673680
const measureDesktopViewport = (tab: ServerTab) =>
674681
(tab.desktopViewportNext ??= tab.desktopViewportRead.then(() => {
675682
tab.desktopViewportNext = null;
683+
tab.desktopViewportStale = false;
676684
return (tab.desktopViewportRead = readDesktopViewport(tab));
677685
}));
678686

@@ -887,6 +895,8 @@ const make = Effect.gen(function* () {
887895
desktopViewport: null,
888896
desktopViewportRead: Promise.resolve(null),
889897
desktopViewportNext: null,
898+
desktopViewportStale: true,
899+
desktopResize: Promise.withResolvers<void>(),
890900
loading: false,
891901
closing: false,
892902
recording: null,
@@ -904,9 +914,15 @@ const make = Effect.gen(function* () {
904914
if (desktop) {
905915
// The desktop resizes and zooms its webview itself; viewers map input with the new size.
906916
const remeasure = () => {
917+
tab.desktopViewportStale = true;
907918
if (tab.viewers.size > 0) return broadcastViewport(tab).catch(constVoid);
908919
};
909-
cdp.on("Page.frameResized", remeasure);
920+
cdp.on("Page.frameResized", () => {
921+
const drawn = tab.desktopResize;
922+
tab.desktopResize = Promise.withResolvers();
923+
drawn.resolve();
924+
return remeasure();
925+
});
910926
// A read a navigation broke has a fresh page to measure.
911927
cdp.on("Page.loadEventFired", remeasure);
912928
// Closing a dialog resizes nothing, and the host can close it without us.
@@ -1618,7 +1634,12 @@ const make = Effect.gen(function* () {
16181634
};
16191635
}
16201636
const url = ServerBrowserPage.withoutDataPayload(tab.page.url());
1621-
const viewport = tab.desktop ? await layoutSize(tab) : tab.page.viewportSize();
1637+
// Page resizes and loads mark the size stale; only then does status read the page.
1638+
const viewport = tab.desktop
1639+
? tab.desktopViewportStale || tab.dialog
1640+
? await layoutSize(tab)
1641+
: await tab.desktopViewportRead
1642+
: tab.page.viewportSize();
16221643
const status = {
16231644
available: true,
16241645
visible: tab.viewers.size > 0,
@@ -2175,7 +2196,7 @@ const make = Effect.gen(function* () {
21752196
message: "A browser dialog is pending. Read preview_status.",
21762197
reason: "dialogPending",
21772198
});
2178-
return executeTabOperation(tab, request);
2199+
return executeTabOperation(tab, request, deadlineMs);
21792200
});
21802201
}
21812202
return tab.control.agent(
@@ -2191,7 +2212,7 @@ const make = Effect.gen(function* () {
21912212
tab.agentOperating = !READ_OPERATIONS.has(request.operation);
21922213
updateFileChooserInterception(tab);
21932214
try {
2194-
return await executeTabOperation(tab, request);
2215+
return await executeTabOperation(tab, request, deadlineMs);
21952216
} finally {
21962217
tab.agentOperating = false;
21972218
updateFileChooserInterception(tab);
@@ -2202,7 +2223,11 @@ const make = Effect.gen(function* () {
22022223
);
22032224
};
22042225

2205-
const executeTabOperation = async (tab: ServerTab, request: PreviewAutomationRequest) => {
2226+
const executeTabOperation = async (
2227+
tab: ServerTab,
2228+
request: PreviewAutomationRequest,
2229+
deadlineMs: number,
2230+
) => {
22062231
const input = request.input;
22072232
switch (request.operation) {
22082233
case "navigate": {
@@ -2219,14 +2244,38 @@ const make = Effect.gen(function* () {
22192244
}
22202245
case "resize": {
22212246
const setting = resolvePreviewViewport(input as PreviewAutomationResizeInput);
2247+
const from = fixedViewportSize(tab.setting);
2248+
const to = fixedViewportSize(setting);
2249+
const drawn = tab.desktopResize.promise;
22222250
await runPromise(
22232251
manager.resize({ threadId: tab.threadId, tabId: tab.tabId, viewport: setting }),
22242252
);
22252253
await applySetting(tab, setting);
2254+
if (!tab.desktop) {
2255+
return {
2256+
tabId: tab.tabId,
2257+
setting,
2258+
viewport: tab.page.viewportSize() ?? UNATTACHED_FILL_VIEWPORT,
2259+
};
2260+
}
2261+
const changed = from?.width !== to?.width || from?.height !== to?.height;
2262+
const remainingMs = () => Math.max(0, deadlineMs - performance.now());
2263+
const within = <A>(promise: Promise<A>, ms: number) =>
2264+
runPromise(
2265+
Effect.promise(() => promise).pipe(
2266+
Effect.timeoutOption(ms),
2267+
Effect.map(Option.getOrNull),
2268+
),
2269+
);
2270+
// The desktop draws the new size after the session update; its page resize says it has.
2271+
// Half the time left keeps room for the read, which runs even if nothing was drawn,
2272+
// because a page can keep another size than the one asked for.
2273+
if (changed) await within(drawn, Math.min(DESKTOP_RESIZE_WAIT_MS, remainingMs() / 2));
2274+
const measured = await within(layoutSize(tab), remainingMs());
22262275
return {
22272276
tabId: tab.tabId,
22282277
setting,
2229-
viewport: tab.page.viewportSize() ?? UNATTACHED_FILL_VIEWPORT,
2278+
viewport: measured ?? tab.desktopViewport ?? UNATTACHED_FILL_VIEWPORT,
22302279
};
22312280
}
22322281
case "setColorScheme": {
@@ -2345,6 +2394,8 @@ const make = Effect.gen(function* () {
23452394
return;
23462395
}
23472396
if (event.type === "resized") {
2397+
// Desktop tabs only record the setting, so status need not wait behind an agent action.
2398+
if (tab.desktop) tab.setting = event.snapshot.viewport ?? FILL_PREVIEW_VIEWPORT;
23482399
await tab.control
23492400
.system(async () => {
23502401
if (

0 commit comments

Comments
 (0)