Skip to content

Commit 67e6894

Browse files
donjor-agent[bot]donjor
authored andcommitted
feat(desktop): duck selected speakers during dictation (#1687)
## Problem Speaker playback can overwhelm dictation; recording needs configurable ducking and restoration after capture ends. ## Change Adds Linux desktop output selection with Select all, a relative target volume, independent fade durations, and restoration across stop, cancel, errors, window closure, and shutdown while preserving manual volume changes. **Controls:** checkbox dropdown, Lucide refresh icon, synchronized sliders and numeric inputs **Hardware:** accepted-volume readback handles speakers that round commands **Lifecycle:** upload and live dictation share recording cleanup ## Scope and approval The fork maintainer requested the feature, authorized physical speaker testing, confirmed restoration, and requested this PR in the implementation thread. **Scope:** desktop-local settings and outputs; browser/mobile clients do not control host speakers; unsupported desktop platforms report availability ## Verification **Head:** `c78ce88788215acbf1f664fd9cdfa4bac66634c1` ### Checks - [x] **CI:** eight required checks passed - [x] **Fork:** `vp run fork:ci`, exit 0 - [x] **Tests:** 22 focused regressions passed - [x] **Desktop:** typecheck, exit 0 - [x] **Build:** `vp run build:desktop`, exit 0 - [ ] **Aftercare:** no current-head review result **Tests:** ducking, audio adapter, service shutdown, and desktop lifecycle **Build:** includes the web production build **Earlier checks:** web/contracts typechecks, exit 0 before the desktop shutdown fix ### Recording and restoration - [x] **Controls:** selection and slider/input sync - [x] **Cancel:** G560 restored on `c78ce88` - [x] **Quit:** restored before Electron exited - [x] **Stop:** G560 restored on `7e1f33b` **Defaults tested:** 20% of starting volume, instant ducking, 1000 ms restoration **G560 Cancel on current head:** `pactl get-sink-volume`, both channels ```text Before: 36700 / 56% Recording: 7864 / 12% Mid restoration: 20972 / 32% Restored: 36700 / 56% ``` **Window-close/quit on current head:** both channels restored from `7864` to original `36700` before Electron exited **Regressions:** immediate and midway stop-then-close; blocked volume write prevents shutdown completion ### UI evidence **Initial control refinement:** output checklist ![Initial output checklist and text refresh button](https://github.com/user-attachments/assets/e431a4e7-df10-45da-baa4-d4feac878777) **Output selection:** checkbox dropdown with Select all ![Output dropdown with Select all and device checkboxes](https://github.com/user-attachments/assets/1adc1b16-d904-4b1d-a83f-aa6707aa76dc) **Volume and timing:** independent sliders and unit-labelled inputs ![Volume and fade sliders with percent and millisecond inputs](https://github.com/user-attachments/assets/c0c100c8-e53b-4990-8278-fd4d50176ed3) **Clip:** start/stop a silent capture; expected no-speech message **Provenance:** `0b92f75`, before cleanup/shutdown fixes; controls unchanged; current-head hardware measurements above **Timing:** original speed; final 7.17 seconds of idle footage trimmed; speaker fade measured with `pactl` https://github.com/user-attachments/assets/1be26ed4-4df3-47db-b3ea-fa65ed7f7902 Model: GPT-6.1-Sol. Harness: Codex. ## Fork trailers Squashes: - e4fcd28 feat(desktop): duck selected speakers during dictation (#1687) (#1687) - fb2bbd4 fixup! feat(desktop): duck selected speakers during dictation (#1687) (#1687) - 72b31ee feat(desktop): duck selected speakers during dictation (#1687) (#1687) - 43a2a14 fixup! feat(desktop): duck selected speakers during dictation (#1687) (#1687) Fork-Domain: voice-input Fork-Tier: core Fork-Upstreamable: no Co-authored-by: donjor <[email protected]>
1 parent a47c10c commit 67e6894

24 files changed

Lines changed: 1360 additions & 3 deletions

‎apps/desktop/src/app/DesktopApp.ts‎

Lines changed: 2 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -45,6 +45,7 @@ import * as DesktopSnapShot from "../snapShot/DesktopSnapShot.ts";
4545
import * as DesktopWslBackend from "../wsl/DesktopWslBackend.ts";
4646
import * as DesktopRendererHistory from "../telemetry/DesktopRendererHistory.ts";
4747
import * as HostProcess from "@t3tools/shared/HostProcess";
48+
import * as DesktopVoiceDuckingFork from "../voice-input/DesktopVoiceDucking.fork.ts"; // fork-hook: voice-input/ducking-shutdown-import
4849

4950
const DEFAULT_DESKTOP_BACKEND_PORT = 3773;
5051
const MAX_TCP_PORT = 65_535;
@@ -473,6 +474,7 @@ const scopedProgram = Effect.scoped(
473474
}).pipe(Effect.ensuring(rendererHistory.shutdown), Effect.ensuring(shutdown.markComplete)), // fork-hook: backend-attach/client-only-finalizer
474475
);
475476

477+
yield* DesktopVoiceDuckingFork.registerShutdown; // fork-hook: voice-input/ducking-program-finalizer
476478
yield* startup;
477479
yield* shutdown.awaitRequest;
478480
}),

‎apps/desktop/src/ipc/DesktopIpcHandlers.ts‎

Lines changed: 4 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -2,6 +2,7 @@ import * as Effect from "effect/Effect";
22

33
import { receiveProviderAuthCallback, cancelProviderAuthCallback } from "./methods/providerAuth.ts";
44
import * as DesktopIpc from "./DesktopIpc.ts";
5+
import * as VoiceDuckingFork from "./methods/voiceDucking.fork.ts"; // fork-hook: voice-input/ducking-ipc-import
56
import { installNotificationBadge } from "./methods/notificationBadge.ts";
67
import { getClientSettings, setClientSettings } from "./methods/clientSettings.ts";
78
import { getBackendModeState, setBackendMode } from "./methods/backendMode.ts";
@@ -91,6 +92,9 @@ import {
9192

9293
export const installDesktopIpcHandlers = Effect.fn("desktop.ipc.installHandlers")(function* () {
9394
const ipc = yield* DesktopIpc.DesktopIpc;
95+
yield* ipc.handle(VoiceDuckingFork.listOutputs); // fork-hook: voice-input/ducking-outputs-ipc
96+
yield* ipc.handle(VoiceDuckingFork.start); // fork-hook: voice-input/ducking-start-ipc
97+
yield* ipc.handle(VoiceDuckingFork.stop); // fork-hook: voice-input/ducking-stop-ipc
9498
yield* installNotificationBadge();
9599
yield* PreviewIpc.installPreviewEventForwarding();
96100

Lines changed: 50 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,50 @@
1+
import { VoiceDuckingOutputsFork, VoiceDuckingStartFork } from "@t3tools/contracts";
2+
import * as Effect from "effect/Effect";
3+
import * as Option from "effect/Option";
4+
import * as Schema from "effect/Schema";
5+
import { ElectronWindow } from "../../electron/ElectronWindow.ts";
6+
import {
7+
DesktopVoiceDuckingFork,
8+
VoiceDuckingErrorFork,
9+
} from "../../voice-input/DesktopVoiceDucking.fork.ts";
10+
import * as Channels from "../../voice-input/channels.fork.ts";
11+
import * as DesktopIpc from "../DesktopIpc.ts";
12+
import { resolveRegisteredSenderWindow } from "./snapShotSender.fork.ts";
13+
14+
const owner = Effect.fn(function* (event?: DesktopIpc.DesktopIpcInvokeEvent) {
15+
const windows = yield* ElectronWindow;
16+
if (!event || Option.isNone(yield* resolveRegisteredSenderWindow(windows, event))) {
17+
return yield* new VoiceDuckingErrorFork({
18+
message: "Speaker volume requires a registered desktop window.",
19+
});
20+
}
21+
return event.sender.id;
22+
});
23+
24+
export const listOutputs = DesktopIpc.makeIpcMethod({
25+
channel: Channels.VOICE_DUCKING_OUTPUTS_FORK,
26+
payload: Schema.Void,
27+
result: VoiceDuckingOutputsFork,
28+
handler: Effect.fn(function* (_input, event) {
29+
yield* owner(event);
30+
return yield* (yield* DesktopVoiceDuckingFork).list;
31+
}),
32+
});
33+
export const start = DesktopIpc.makeIpcMethod({
34+
channel: Channels.VOICE_DUCKING_START_FORK,
35+
payload: VoiceDuckingStartFork,
36+
result: Schema.Void,
37+
handler: Effect.fn(function* ({ sessionId, settings }, event) {
38+
const sender = yield* owner(event);
39+
yield* (yield* DesktopVoiceDuckingFork).start(sender, sessionId, settings);
40+
}),
41+
});
42+
export const stop = DesktopIpc.makeIpcMethod({
43+
channel: Channels.VOICE_DUCKING_STOP_FORK,
44+
payload: Schema.Struct({ sessionId: VoiceDuckingStartFork.fields.sessionId }),
45+
result: Schema.Void,
46+
handler: Effect.fn(function* ({ sessionId }, event) {
47+
const sender = yield* owner(event);
48+
yield* (yield* DesktopVoiceDuckingFork).stop(sender, sessionId);
49+
}),
50+
});

‎apps/desktop/src/main.ts‎

Lines changed: 2 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -23,6 +23,7 @@ import type { RemoteT3RunnerOptions } from "@t3tools/ssh/tunnel";
2323
import serverPackageJson from "../../server/package.json" with { type: "json" };
2424

2525
import * as DesktopIpc from "./ipc/DesktopIpc.ts";
26+
import * as DesktopVoiceDuckingFork from "./voice-input/DesktopVoiceDucking.fork.ts"; // fork-hook: voice-input/ducking-layer-import
2627
import * as ElectronApp from "./electron/ElectronApp.ts";
2728
import * as ElectronDialog from "./electron/ElectronDialog.ts";
2829
import * as ElectronMenu from "./electron/ElectronMenu.ts";
@@ -145,6 +146,7 @@ const layerElectron = Layer.mergeAll(
145146
);
146147

147148
const layerDesktopFoundation = Layer.mergeAll(
149+
DesktopVoiceDuckingFork.layer, // fork-hook: voice-input/ducking-layer
148150
MacPermissions.layer,
149151
DesktopState.layer,
150152
DesktopShutdown.layer,

‎apps/desktop/src/preload.ts‎

Lines changed: 2 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -11,6 +11,7 @@ import { exposeClerkBridge } from "@clerk/electron/preload";
1111
import { contextBridge, ipcRenderer, webFrame, webUtils } from "electron";
1212

1313
import * as IpcChannels from "./ipc/channels.ts";
14+
import { makeVoiceDuckingBridgeFork } from "./voice-input/preload.fork.ts"; // fork-hook: voice-input/ducking-preload-import
1415
import { mergeLegacyLocalStorage } from "./legacyLocalStorageMerge.ts";
1516
import { makeAttachedPrimaryBridge } from "./attachedPrimaryPreload.fork.ts"; // fork-hook: backend-attach/preload-import
1617
import { exposePreviewCapability } from "./preview/WindowPolicy.preload.ts";
@@ -75,6 +76,7 @@ function unwrapEnsureSshEnvironmentResult(result: unknown) {
7576
}
7677

7778
const desktopBridge = {
79+
voiceDuckingFork: makeVoiceDuckingBridgeFork(ipcRenderer), // fork-hook: voice-input/ducking-preload
7880
getAppBranding: () => {
7981
const result = ipcRenderer.sendSync(IpcChannels.GET_APP_BRANDING_CHANNEL);
8082
if (typeof result !== "object" || result === null) {
Lines changed: 115 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,115 @@
1+
import { assert, it } from "@effect/vitest";
2+
import { vi } from "vite-plus/test";
3+
import { DEFAULT_VOICE_DUCKING_SETTINGS_FORK } from "@t3tools/contracts";
4+
import * as Deferred from "effect/Deferred";
5+
import * as Effect from "effect/Effect";
6+
import * as Fiber from "effect/Fiber";
7+
import * as Layer from "effect/Layer";
8+
import type * as Electron from "electron";
9+
import { DesktopEnvironment } from "../app/DesktopEnvironment.ts";
10+
import * as DesktopLifecycle from "../app/DesktopLifecycle.ts";
11+
import * as DesktopShutdown from "../app/DesktopShutdown.ts";
12+
import * as DesktopState from "../app/DesktopState.ts";
13+
import { ElectronApp } from "../electron/ElectronApp.ts";
14+
import { ElectronTheme } from "../electron/ElectronTheme.ts";
15+
import { ElectronWindow } from "../electron/ElectronWindow.ts";
16+
import { DesktopWindow } from "../window/DesktopWindow.ts";
17+
import * as DesktopVoiceDucking from "./DesktopVoiceDucking.fork.ts";
18+
19+
const audio = vi.hoisted(() => ({ list: vi.fn(), setVolumes: vi.fn() }));
20+
vi.mock("./audioOutputs.fork.ts", () => ({ linuxAudioOutputsFork: audio }));
21+
vi.mock("electron", () => ({
22+
webContents: {
23+
fromId: () => ({
24+
isDestroyed: () => false,
25+
once: vi.fn(),
26+
on: vi.fn(),
27+
removeListener: vi.fn(),
28+
}),
29+
},
30+
}));
31+
32+
it.effect("holds Electron quit until pending speaker restoration completes", () =>
33+
Effect.gen(function* () {
34+
const original = [60000, 30000];
35+
let volumes = original;
36+
const restoring = Promise.withResolvers<void>();
37+
const allowRestore = Promise.withResolvers<void>();
38+
const ready = yield* Deferred.make<void>();
39+
const quit = yield* Deferred.make<void>();
40+
const listeners = new Map<string, (...args: readonly unknown[]) => void>();
41+
audio.list.mockImplementation(async () => [
42+
{ id: "speakers", label: "Speakers", isDefault: true, volumes: [...volumes] },
43+
]);
44+
audio.setVolumes.mockImplementation(async (_id: string, next: readonly number[]) => {
45+
if (next[0] === original[0]) {
46+
restoring.resolve();
47+
await allowRestore.promise;
48+
}
49+
volumes = [...next];
50+
return volumes;
51+
});
52+
const environment = Layer.succeed(DesktopEnvironment, {
53+
platform: "linux",
54+
} as DesktopEnvironment["Service"]);
55+
const layer = Layer.mergeAll(
56+
DesktopVoiceDucking.layer.pipe(Layer.provide(environment)),
57+
DesktopLifecycle.layer,
58+
DesktopShutdown.layer,
59+
DesktopState.layer,
60+
environment,
61+
Layer.mock(ElectronApp)({
62+
on: (event, listener) =>
63+
Effect.sync(() => {
64+
listeners.set(event, listener as unknown as (...args: readonly unknown[]) => void);
65+
}),
66+
onBeforeQuitForUpdate: () => Effect.void,
67+
quit: Effect.sync(() => assert.deepEqual(volumes, original)).pipe(
68+
Effect.andThen(Deferred.succeed(quit, undefined)),
69+
Effect.asVoid,
70+
),
71+
}),
72+
Layer.mock(ElectronTheme)({ onUpdated: () => Effect.void }),
73+
Layer.mock(ElectronWindow)({ destroyAll: Effect.void }),
74+
Layer.mock(DesktopWindow)({ flushMainWindowBounds: Effect.void }),
75+
);
76+
yield* Effect.scoped(
77+
Effect.gen(function* () {
78+
const shutdown = yield* DesktopShutdown.DesktopShutdown;
79+
const ducking = yield* DesktopVoiceDucking.DesktopVoiceDuckingFork;
80+
yield* (yield* DesktopLifecycle.DesktopLifecycle).register;
81+
yield* ducking.start(1, "recording", {
82+
...DEFAULT_VOICE_DUCKING_SETTINGS_FORK,
83+
enabled: true,
84+
});
85+
const program = yield* Effect.forkChild(
86+
Effect.scoped(
87+
Effect.gen(function* () {
88+
yield* Effect.addFinalizer(() => shutdown.markComplete);
89+
yield* DesktopVoiceDucking.registerShutdown;
90+
yield* Deferred.succeed(ready, undefined);
91+
yield* shutdown.awaitRequest;
92+
}),
93+
),
94+
);
95+
yield* Deferred.await(ready);
96+
let prevented = false;
97+
listeners.get("before-quit")!({
98+
preventDefault: () => {
99+
prevented = true;
100+
},
101+
} as Electron.Event);
102+
yield* Effect.promise(() => restoring.promise);
103+
assert.isTrue(prevented);
104+
assert.isFalse(yield* shutdown.isComplete);
105+
assert.isFalse(yield* Deferred.isDone(quit));
106+
assert.deepEqual(volumes, [12000, 6000]);
107+
allowRestore.resolve();
108+
yield* Deferred.await(quit);
109+
yield* Fiber.join(program);
110+
assert.isTrue(yield* shutdown.isComplete);
111+
assert.deepEqual(volumes, original);
112+
}),
113+
).pipe(Effect.provide(layer));
114+
}),
115+
);
Lines changed: 128 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,128 @@
1+
import type { VoiceDuckingOutputsFork, VoiceDuckingSettingsFork } from "@t3tools/contracts";
2+
import * as Context from "effect/Context";
3+
import * as Effect from "effect/Effect";
4+
import * as Layer from "effect/Layer";
5+
import * as Schema from "effect/Schema";
6+
import * as Electron from "electron";
7+
import { DesktopEnvironment } from "../app/DesktopEnvironment.ts";
8+
import { linuxAudioOutputsFork } from "./audioOutputs.fork.ts";
9+
import { SpeakerDuckingFork } from "./ducking.fork.ts";
10+
11+
export class VoiceDuckingErrorFork extends Schema.TaggedError<VoiceDuckingErrorFork>()(
12+
"VoiceDuckingErrorFork",
13+
{ message: Schema.String },
14+
) {}
15+
const attempt = <A>(run: () => Promise<A>) =>
16+
Effect.tryPromise({
17+
try: run,
18+
catch: (error) =>
19+
new VoiceDuckingErrorFork({
20+
message: error instanceof Error ? error.message : "Speaker ducking failed.",
21+
}),
22+
});
23+
24+
export class DesktopVoiceDuckingFork extends Context.Service<
25+
DesktopVoiceDuckingFork,
26+
{
27+
readonly list: Effect.Effect<VoiceDuckingOutputsFork>;
28+
readonly start: (
29+
owner: number,
30+
sessionId: string,
31+
settings: VoiceDuckingSettingsFork,
32+
) => Effect.Effect<void, VoiceDuckingErrorFork>;
33+
readonly stop: (owner: number, sessionId: string) => Effect.Effect<void, VoiceDuckingErrorFork>;
34+
readonly shutdown: Effect.Effect<void>;
35+
}
36+
>()("@t3tools/desktop/voice-input/DesktopVoiceDucking.fork/DesktopVoiceDuckingFork") {}
37+
38+
export const layer = Layer.effect(
39+
DesktopVoiceDuckingFork,
40+
Effect.gen(function* () {
41+
const environment = yield* DesktopEnvironment;
42+
const unsupported =
43+
environment.platform === "linux"
44+
? null
45+
: "Speaker ducking currently supports Linux with PipeWire or PulseAudio.";
46+
const runBackground = Effect.runForkWith(yield* Effect.context<never>());
47+
const warn = (error: unknown) => {
48+
runBackground(Effect.logWarning("Speaker ducking:", error));
49+
};
50+
const manager = new SpeakerDuckingFork(linuxAudioOutputsFork, warn);
51+
const watchers = new Map<number, () => void>();
52+
const watchOwner = (id: number) => {
53+
if (watchers.has(id)) return;
54+
const contents = Electron.webContents.fromId(id);
55+
if (!contents || contents.isDestroyed())
56+
throw new Error("Recording window is no longer available.");
57+
const release = () => {
58+
void manager.stop(id).catch(warn);
59+
};
60+
const navigation = (
61+
_event: Electron.Event,
62+
_url: string,
63+
_inPlace: boolean,
64+
mainFrame: boolean,
65+
) => {
66+
if (mainFrame) release();
67+
};
68+
const cleanup = () => {
69+
contents.removeListener("destroyed", destroyed);
70+
contents.removeListener("render-process-gone", release);
71+
contents.removeListener("did-start-navigation", navigation);
72+
watchers.delete(id);
73+
};
74+
const destroyed = () => {
75+
release();
76+
cleanup();
77+
};
78+
contents.once("destroyed", destroyed);
79+
contents.on("render-process-gone", release);
80+
contents.on("did-start-navigation", navigation);
81+
watchers.set(id, cleanup);
82+
};
83+
const shutdown = attempt(async () => {
84+
for (const cleanup of watchers.values()) cleanup();
85+
await manager.close();
86+
}).pipe(
87+
Effect.catch((error) =>
88+
Effect.logWarning("Could not restore speaker volume on shutdown.", error),
89+
),
90+
);
91+
yield* Effect.addFinalizer(() => shutdown);
92+
return DesktopVoiceDuckingFork.of({
93+
list: unsupported
94+
? Effect.succeed({ unavailableReason: unsupported, outputs: [] })
95+
: attempt(async () => ({
96+
unavailableReason: null,
97+
outputs: (await linuxAudioOutputsFork.list()).map(({ id, label, isDefault }) => ({
98+
id,
99+
label,
100+
isDefault,
101+
})),
102+
})).pipe(
103+
Effect.catch((error) =>
104+
Effect.succeed({ unavailableReason: error.message, outputs: [] }),
105+
),
106+
),
107+
start: Effect.fn("DesktopVoiceDuckingFork.start")(function (owner, sessionId, settings) {
108+
return attempt(async () => {
109+
if (unsupported) throw new Error(unsupported);
110+
watchOwner(owner);
111+
await manager.start(owner, sessionId, settings);
112+
if (Electron.webContents.fromId(owner)?.isDestroyed() !== false)
113+
await manager.stop(owner);
114+
});
115+
}),
116+
stop: Effect.fn("DesktopVoiceDuckingFork.stop")(function (owner, sessionId) {
117+
return attempt(() => manager.stop(owner, sessionId));
118+
}),
119+
shutdown,
120+
});
121+
}),
122+
);
123+
124+
/** Register after the completion finalizer so restoration runs before Electron may quit. */
125+
export const registerShutdown = Effect.gen(function* () {
126+
const ducking = yield* DesktopVoiceDuckingFork;
127+
yield* Effect.addFinalizer(() => ducking.shutdown);
128+
});

0 commit comments

Comments
 (0)