Skip to content

Commit 7b11c61

Browse files
wesbillmanCarlPrincess DonutMongoMordecai
authored
test: fix recurring CI flakes in desktop and Postgres lanes (block#8343)
Fixes the CI flakes that failed most often across `main` and unrelated branches from Oct 5 to Oct 14. The counts come from 150 failed `CI` runs. Every change except the nextest config is test-only. | Flake | Runs hit | Root cause | Fix | |---|---|---|---| | `probe_codex_acp_version_…descendant_holds_pipe_open` | 5 | The test writes its script in-process and then execs it. If a sibling test spawns a process in that window, the child inherits the open write fd and Linux rejects the exec with `ETXTBSY`. The probe swallows that error as `None`. | Write probe scripts from a short-lived `/bin/sh` (`write_executable_script`) | | `empty-edit-delete.spec.ts` (`:51`, `:74`, `:95`) | 11 | The seeded `mock-general-welcome` id isn't hex, so `formatTimelineMessages` can't overlay its edit. The post-subscribe window refetch then reverts the row. | Emit a per-test message with a real id after the exact channel subscription is live | | `huddle-transcription.spec.ts:798` | 9 | The composer's delayed initial autofocus (TipTap `focus()` runs in rAF) lands after the voice popover opens and dismisses it. | Wait for `message-input` focus before opening the popover | | `voice-note.spec.ts:906` | 6 | The fixture `ping.mp3` is about 1s long, so playback can end before `paused === false` is observed. | Loop the fixture audio | | `workflows.spec.ts:291` | 5 | The `Message text` fill runs before the inspector swaps to the new step, so it fills the trigger's condition instead. | Scope the fill to the step inspector after its `Action` menu appears | | `cluster_global_probe_commits_tokens_and_sessions_prove_coverage` | 3 | The `max-threads = 1` group only serializes these tests with each other. Other tests' sessions still mask cluster-wide `pg_stat_activity` (`MaskedActivity`). | `threads-required = "num-test-threads"` | | `test_usage_metrics_lock_has_single_owner_and_releases_on_drop` | 1 (this PR) | The test retries the lock right after dropping the leader. Postgres releases a session advisory lock only after the backend notices the disconnect, so the retry can run first. | Poll for the successor, bounded at 5s | ### Validation - **ETXTBSY:** a Linux experiment with Rust 1.95 using the same write→chmod→spawn pattern under concurrent spawns hit `ETXTBSY` 4–7% of the time. Using the `sh` writer, it hit 0 of 6,000. I did not catch it inside the full suite locally, so this fix is based on the mechanism, not an in-suite repro. All 11 probe tests pass on Linux and macOS. - **empty-edit-delete:** the baseline failed 16 of 45 runs (`--repeat-each`, 6 workers, no retries). The spec fix alone brings that to 4/45. The remaining failures come from a separate product race, described below. - **huddle:** the baseline failed 5 of 10 runs. With the fix, 60 of 60 passed without retries. - **nextest:** checked with a throwaway crate on 0.9.136. A grouped test with `num-test-threads` ran alone, even with `max-threads = 1`. - I have not stress-run the voice-note and workflows fixes; they rely on CI. ### Not in this PR - **Product bug in `useChannelLinks`:** the debounced `#channel` autocomplete close lets Enter accept a stale suggestion for about 120ms. That accounts for the remaining `empty-edit-delete` failures, and it affects users too. With the spec fix plus a synchronous close, the spec passed 120 of 120 runs. This needs a product decision, so it's held for a separate PR. - **Composer autofocus race:** the delayed autofocus can close an overlay the user just opened. The real fix is to revalidate when the delayed focus runs. Overlaps block#6530. - **`thread-head-stale-edit.spec.ts:55`:** a forced click on a pointer-events-none action bar is suspected, but I couldn't reproduce it. - **Remaining ETXTBSY exposure:** about 40 other Rust tests write and then exec scripts, and 17 specs still edit the non-hex seed message. Root-cause lanes by Princess Donut, Mongo and Mordecai. Carl integrated and verified them. 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Signed-off-by: Wes <[email protected]> Co-authored-by: Carl <32a2e2c9d428ee08902cab75d956da2c1d235a22d4766b0dd4138bf6e2e5db1d@buzz.block.builderlab.xyz> Co-authored-by: Princess Donut <5d97ac8c272fa949af56c71e586146b1706e5c4c3daa125db5b4625e12e0686c@buzz.block.builderlab.xyz> Co-authored-by: Mongo <81cabd2ca1792494372ed410283c00770c0cf35afca052e940fa522883212b45@buzz.block.builderlab.xyz> Co-authored-by: Mordecai <f314b6033b6e13df94c8e9e1187d3666b6b9ce54e66ee986e31cdaa909e8dcbc@buzz.block.builderlab.xyz> Co-authored-by: Claude Opus 5.5 <[email protected]>
1 parent 8222e99 commit 7b11c61

8 files changed

Lines changed: 113 additions & 60 deletions

File tree

‎.config/nextest.toml‎

Lines changed: 4 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -32,6 +32,10 @@ max-threads = 1
3232
[[profile.postgres-ci.overrides]]
3333
filter = "test(/cluster_global_/)"
3434
test-group = "postgres-cluster-global"
35+
# The group only serializes these tests with each other. They read unfiltered,
36+
# cluster-wide `pg_stat_activity`, so any overlapping test's session can mask
37+
# activity; claim every worker so they run alone.
38+
threads-required = "num-test-threads"
3539

3640
[[profile.postgres-ci.scripts]]
3741
# Script filters are separate from default-filter: they attach the setup and

‎crates/buzz-db/src/store/usage.rs‎

Lines changed: 14 additions & 4 deletions
Original file line numberDiff line numberDiff line change
@@ -928,14 +928,24 @@ mod postgres_tests {
928928
);
929929

930930
drop(leader);
931-
assert!(
932-
second
931+
// Dropping closes the socket; the server releases the session lock only
932+
// once its backend observes the disconnect, so poll briefly.
933+
let deadline = tokio::time::Instant::now() + std::time::Duration::from_secs(5);
934+
let successor = loop {
935+
let attempt = second
933936
.try_lock_usage_metrics(key)
934937
.await
935-
.expect("lock attempt after leader drop")
936-
.is_some(),
938+
.expect("lock attempt after leader drop");
939+
if attempt.is_some() || tokio::time::Instant::now() >= deadline {
940+
break attempt;
941+
}
942+
tokio::time::sleep(std::time::Duration::from_millis(20)).await;
943+
};
944+
assert!(
945+
successor.is_some(),
937946
"dropping the detached session releases its advisory lock"
938947
);
948+
drop(successor);
939949

940950
// Release any remaining session state before DROP DATABASE.
941951
drop(first);

‎desktop/src-tauri/src/managed_agents/discovery/tests.rs‎

Lines changed: 31 additions & 37 deletions
Original file line numberDiff line numberDiff line change
@@ -612,21 +612,36 @@ fn update_time_override_preserves_pin_for_persona_less_agent() {
612612
mod effort_clear;
613613
mod forced_discovery;
614614
mod managed_path_resolution;
615+
616+
/// Write an executable script without opening it for writing in this process.
617+
///
618+
/// Linux refuses to exec a file while any process holds it open for writing
619+
/// (`ETXTBSY`). A sibling test that spawns between our open and close copies
620+
/// that descriptor into its child until the child execs, so an in-process
621+
/// `fs::write` can make the probe's spawn fail intermittently. A short-lived
622+
/// shell writes the file instead, and its descriptor is gone once it exits.
623+
#[cfg(unix)]
624+
fn write_executable_script(path: &std::path::Path, contents: impl AsRef<str>) {
625+
let status = std::process::Command::new("/bin/sh")
626+
.args(["-c", r#"printf '%s' "$2" > "$1" && chmod 755 "$1""#, "sh"])
627+
.arg(path)
628+
.arg(contents.as_ref())
629+
.status()
630+
.expect("spawn script writer");
631+
assert!(status.success(), "write script {}", path.display());
632+
}
633+
615634
#[cfg(unix)]
616635
#[test]
617636
fn probe_codex_acp_version_parses_full_semver_output() {
618-
use std::os::unix::fs::PermissionsExt;
619-
620637
// Simulate a current `@agentclientprotocol/codex-acp` output.
621638
let dir = std::env::temp_dir().join(format!("buzz-probe-1x-{}", uuid::Uuid::new_v4()));
622639
std::fs::create_dir_all(&dir).expect("create temp dir");
623640
let bin = dir.join("codex-acp");
624-
std::fs::write(
641+
write_executable_script(
625642
&bin,
626643
"#!/bin/sh\necho '@agentclientprotocol/codex-acp 1.1.7'\nexit 0\n",
627-
)
628-
.expect("write script");
629-
std::fs::set_permissions(&bin, std::fs::Permissions::from_mode(0o755)).expect("chmod script");
644+
);
630645

631646
let version = probe_codex_acp_version(&bin);
632647
let _ = std::fs::remove_dir_all(dir);
@@ -643,14 +658,11 @@ mod codex_version;
643658
#[cfg(unix)]
644659
#[test]
645660
fn probe_codex_acp_version_returns_none_for_nonzero_exit() {
646-
use std::os::unix::fs::PermissionsExt;
647-
648661
// Simulate old 0.16.x adapter: `--version` is unrecognised, exits non-zero
649662
let dir = std::env::temp_dir().join(format!("buzz-probe-0x-{}", uuid::Uuid::new_v4()));
650663
std::fs::create_dir_all(&dir).expect("create temp dir");
651664
let bin = dir.join("codex-acp");
652-
std::fs::write(&bin, "#!/bin/sh\nexit 1\n").expect("write script");
653-
std::fs::set_permissions(&bin, std::fs::Permissions::from_mode(0o755)).expect("chmod script");
665+
write_executable_script(&bin, "#!/bin/sh\nexit 1\n");
654666

655667
let version = probe_codex_acp_version(&bin);
656668
let _ = std::fs::remove_dir_all(dir);
@@ -677,17 +689,13 @@ fn probe_codex_acp_version_returns_none_for_missing_binary() {
677689
#[cfg(unix)]
678690
#[test]
679691
fn codex_adapter_availability_available_for_minimum_supported_binary() {
680-
use std::os::unix::fs::PermissionsExt;
681-
682692
let dir = std::env::temp_dir().join(format!("buzz-avail-1x-{}", uuid::Uuid::new_v4()));
683693
std::fs::create_dir_all(&dir).expect("create temp dir");
684694
let bin = dir.join("codex-acp");
685-
std::fs::write(
695+
write_executable_script(
686696
&bin,
687697
"#!/bin/sh\necho '@agentclientprotocol/codex-acp 1.10.0'\nexit 0\n",
688-
)
689-
.expect("write script");
690-
std::fs::set_permissions(&bin, std::fs::Permissions::from_mode(0o755)).expect("chmod script");
698+
);
691699

692700
let status = codex_adapter_availability(&bin);
693701
let _ = std::fs::remove_dir_all(dir);
@@ -702,14 +710,11 @@ fn codex_adapter_availability_available_for_minimum_supported_binary() {
702710
#[cfg(unix)]
703711
#[test]
704712
fn codex_adapter_availability_outdated_for_0x_binary() {
705-
use std::os::unix::fs::PermissionsExt;
706-
707713
// Simulate old 0.16.x: `--version` exits non-zero (unrecognised flag)
708714
let dir = std::env::temp_dir().join(format!("buzz-avail-0x-{}", uuid::Uuid::new_v4()));
709715
std::fs::create_dir_all(&dir).expect("create temp dir");
710716
let bin = dir.join("codex-acp");
711-
std::fs::write(&bin, "#!/bin/sh\nexit 1\n").expect("write script");
712-
std::fs::set_permissions(&bin, std::fs::Permissions::from_mode(0o755)).expect("chmod script");
717+
write_executable_script(&bin, "#!/bin/sh\nexit 1\n");
713718

714719
let status = codex_adapter_availability(&bin);
715720
let _ = std::fs::remove_dir_all(dir);
@@ -726,18 +731,13 @@ fn codex_adapter_availability_outdated_for_0x_binary() {
726731
#[cfg(unix)]
727732
#[test]
728733
fn codex_adapter_availability_outdated_for_uncomparable_version() {
729-
use std::os::unix::fs::PermissionsExt;
730-
731734
for version in ["1.2", "1.2.0-rc1"] {
732735
let dir = tempfile::tempdir().expect("temp dir");
733736
let bin = dir.path().join("codex-acp");
734-
std::fs::write(
737+
write_executable_script(
735738
&bin,
736739
format!("#!/bin/sh\necho '@agentclientprotocol/codex-acp {version}'\nexit 0\n"),
737-
)
738-
.expect("write script");
739-
std::fs::set_permissions(&bin, std::fs::Permissions::from_mode(0o755))
740-
.expect("chmod script");
740+
);
741741

742742
assert_eq!(
743743
codex_adapter_availability(&bin),
@@ -766,7 +766,6 @@ fn codex_adapter_availability_outdated_for_missing_binary() {
766766
#[cfg(unix)]
767767
#[test]
768768
fn probe_codex_acp_version_returns_none_for_hung_direct_child() {
769-
use std::os::unix::fs::PermissionsExt;
770769
use std::time::Instant;
771770

772771
// Simulate a process that writes version to stdout then blocks forever.
@@ -775,12 +774,10 @@ fn probe_codex_acp_version_returns_none_for_hung_direct_child() {
775774
let dir = std::env::temp_dir().join(format!("buzz-probe-hung-{}", uuid::Uuid::new_v4()));
776775
std::fs::create_dir_all(&dir).expect("create temp dir");
777776
let bin = dir.join("codex-acp");
778-
std::fs::write(
777+
write_executable_script(
779778
&bin,
780779
"#!/bin/sh\nprintf '@agentclientprotocol/codex-acp 1.1.2\\n'\nexec sleep 300\n",
781-
)
782-
.expect("write script");
783-
std::fs::set_permissions(&bin, std::fs::Permissions::from_mode(0o755)).expect("chmod script");
780+
);
784781

785782
let start = Instant::now();
786783
let version = probe_codex_acp_version(&bin);
@@ -801,7 +798,6 @@ fn probe_codex_acp_version_returns_none_for_hung_direct_child() {
801798
#[cfg(unix)]
802799
#[test]
803800
fn probe_codex_acp_version_returns_version_when_descendant_holds_pipe_open() {
804-
use std::os::unix::fs::PermissionsExt;
805801
use std::time::Instant;
806802

807803
// Simulate a process that forks a background child which inherits stdout
@@ -817,12 +813,10 @@ fn probe_codex_acp_version_returns_version_when_descendant_holds_pipe_open() {
817813
let dir = std::env::temp_dir().join(format!("buzz-probe-descendant-{}", uuid::Uuid::new_v4()));
818814
std::fs::create_dir_all(&dir).expect("create temp dir");
819815
let bin = dir.join("codex-acp");
820-
std::fs::write(
816+
write_executable_script(
821817
&bin,
822818
"#!/bin/sh\necho '@agentclientprotocol/codex-acp 1.1.2'\nsleep 60 &\nexit 0\n",
823-
)
824-
.expect("write script");
825-
std::fs::set_permissions(&bin, std::fs::Permissions::from_mode(0o755)).expect("chmod script");
819+
);
826820

827821
let start = Instant::now();
828822
let version = probe_codex_acp_version(&bin);

‎desktop/src-tauri/src/managed_agents/discovery/tests/codex_version.rs‎

Lines changed: 7 additions & 14 deletions
Original file line numberDiff line numberDiff line change
@@ -1,10 +1,11 @@
11
use super::super::probe_codex_acp_version_with_path;
2+
#[cfg(unix)]
3+
use super::write_executable_script;
24

35
#[cfg(unix)]
46
#[test]
57
fn probe_codex_acp_version_uses_augmented_path_for_env_shebang_interpreter() {
68
use std::fs;
7-
use std::os::unix::fs::PermissionsExt;
89
let temp = tempfile::tempdir().expect("temp dir");
910
let script_dir = temp.path().join("script-bin");
1011
let interpreter_dir = temp.path().join("interpreter-bin");
@@ -14,17 +15,13 @@ fn probe_codex_acp_version_uses_augmented_path_for_env_shebang_interpreter() {
1415
fs::create_dir_all(&empty_path_dir).expect("empty path dir");
1516

1617
let interpreter_path = interpreter_dir.join("node");
17-
fs::write(
18+
write_executable_script(
1819
&interpreter_path,
1920
"#!/bin/sh\necho '@agentclientprotocol/codex-acp 1.1.2'\n",
20-
)
21-
.expect("write interpreter");
22-
fs::set_permissions(&interpreter_path, fs::Permissions::from_mode(0o755))
23-
.expect("chmod interpreter");
21+
);
2422

2523
let shim_path = script_dir.join("codex-acp");
26-
fs::write(&shim_path, "#!/usr/bin/env node\n").expect("write shim");
27-
fs::set_permissions(&shim_path, fs::Permissions::from_mode(0o755)).expect("chmod shim");
24+
write_executable_script(&shim_path, "#!/usr/bin/env node\n");
2825

2926
let scrubbed_path = std::env::join_paths([empty_path_dir.as_path()])
3027
.expect("join scrubbed PATH")
@@ -52,18 +49,14 @@ fn probe_codex_acp_version_uses_augmented_path_for_env_shebang_interpreter() {
5249
fn codex_adapter_availability_outdated_for_older_1x_binary() {
5350
use super::super::{codex_adapter_availability, codex_adapter_is_outdated};
5451
use crate::managed_agents::AcpAvailabilityStatus;
55-
use std::os::unix::fs::PermissionsExt;
5652

5753
for version in ["1.1.5", "1.1.7", "1.6.2", "1.9.0"] {
5854
let dir = tempfile::tempdir().expect("temp dir");
5955
let bin = dir.path().join("codex-acp");
60-
std::fs::write(
56+
write_executable_script(
6157
&bin,
6258
format!("#!/bin/sh\necho '@agentclientprotocol/codex-acp {version}'\nexit 0\n"),
63-
)
64-
.expect("write script");
65-
std::fs::set_permissions(&bin, std::fs::Permissions::from_mode(0o755))
66-
.expect("chmod script");
59+
);
6760

6861
assert_eq!(
6962
codex_adapter_availability(&bin),

‎desktop/tests/e2e/empty-edit-delete.spec.ts‎

Lines changed: 41 additions & 4 deletions
Original file line numberDiff line numberDiff line change
@@ -2,11 +2,14 @@ import { expect, test } from "@playwright/test";
22

33
import { installMockBridge } from "../helpers/bridge";
44

5-
// The mock identity's own pre-seeded message in #general (authored by
6-
// DEFAULT_MOCK_IDENTITY.pubkey in e2eBridge.ts). Editing/deleting one's own
5+
// A message the mock identity authors in #general. Editing/deleting one's own
76
// message is exactly Sam's workflow: "delete a message by clearing its edit."
8-
const OWN_MESSAGE_ID = "mock-general-welcome";
9-
const RENDERED_ORIGINAL_CONTENT = "Welcome to general";
7+
// It is emitted per test rather than reusing the seeded `mock-general-welcome`:
8+
// edits and deletions only overlay a target with a real 64-hex event id, so
9+
// that seed keeps its edit only until the post-subscribe window refetch.
10+
const ORIGINAL_CONTENT = "Ship notes for #general";
11+
const RENDERED_ORIGINAL_CONTENT = "Ship notes for general";
12+
let OWN_MESSAGE_ID: string;
1013

1114
// Open the more-actions menu for a message row and wait for the menu to mount.
1215
async function openMoreActionsMenu(
@@ -46,6 +49,40 @@ test.beforeEach(async ({ page }) => {
4649
await page.goto("/");
4750
await page.getByTestId("channel-general").click();
4851
await expect(page.getByTestId("chat-title")).toHaveText("general");
52+
await expect
53+
.poll(() =>
54+
page.evaluate(
55+
() =>
56+
(
57+
window as Window & {
58+
__BUZZ_E2E_HAS_MOCK_LIVE_SUBSCRIPTION__?: (input: {
59+
channelName: string;
60+
kind: number;
61+
exactChannel: boolean;
62+
}) => boolean;
63+
}
64+
).__BUZZ_E2E_HAS_MOCK_LIVE_SUBSCRIPTION__?.({
65+
channelName: "general",
66+
kind: 9,
67+
exactChannel: true,
68+
}) ?? false,
69+
),
70+
)
71+
.toBe(true);
72+
const emitted = await page.evaluate(
73+
(content) =>
74+
(
75+
window as Window & {
76+
__BUZZ_E2E_EMIT_MOCK_MESSAGE__?: (input: {
77+
channelName: string;
78+
content: string;
79+
}) => { id: string } | undefined;
80+
}
81+
).__BUZZ_E2E_EMIT_MOCK_MESSAGE__?.({ channelName: "general", content }),
82+
ORIGINAL_CONTENT,
83+
);
84+
if (!emitted?.id) throw new Error("mock message emit did not return an id");
85+
OWN_MESSAGE_ID = emitted.id;
4986
});
5087

5188
test("clearing an edit to empty prompts to delete, then deletes on confirm", async ({

‎desktop/tests/e2e/huddle-transcription.spec.ts‎

Lines changed: 4 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -850,6 +850,10 @@ test("assigns distinct agent voices and exposes compact per-agent controls", asy
850850
});
851851
expect(new Set(assignedVoices).size).toBe(2);
852852

853+
// Let the huddle chat finish mounting and autofocus before opening an
854+
// overlay; a late composer focus would dismiss the popover.
855+
await expect(page.getByTestId("message-input")).toBeFocused();
856+
853857
await page.getByRole("button", { name: "Voice settings for alice" }).click();
854858
await waitForAnimations(page);
855859
const voiceMenu = page.locator(

‎desktop/tests/e2e/voice-note.spec.ts‎

Lines changed: 5 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -943,6 +943,11 @@ test("starting a duplicate voice-note player pauses the other instance", async (
943943

944944
const cards = page.getByTestId("audio-message-attachment");
945945
await expect(cards).toHaveCount(2);
946+
// The ping fixture lasts about one second, so it can finish before the
947+
// playing state is observed. Loop it so only exclusivity pauses a player.
948+
await cards.locator("audio").evaluateAll((audios) => {
949+
for (const audio of audios) (audio as HTMLAudioElement).loop = true;
950+
});
946951
const firstAudio = cards.nth(0).locator("audio");
947952
const secondAudio = cards.nth(1).locator("audio");
948953

‎desktop/tests/e2e/workflows.spec.ts‎

Lines changed: 7 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -96,7 +96,13 @@ async function createWorkflow(
9696

9797
await dialog.getByRole("button", { name: "Add step", exact: true }).click();
9898
await page.getByRole("menuitem", { name: "Send Message" }).click();
99-
await dialog.getByLabel("Message text").fill("Workflow notification");
99+
// The inspector swaps from the trigger to the new step after the click; an
100+
// unscoped fill can land in the trigger's message-text condition instead.
101+
const stepInspector = dialog.getByTestId("workflow-node-inspector");
102+
await expect(
103+
stepInspector.getByRole("button", { name: "Action", exact: true }),
104+
).toBeVisible();
105+
await stepInspector.getByLabel("Message text").fill("Workflow notification");
100106
if (options?.stepName) {
101107
await dialog.getByRole("button", { name: "Step details" }).click();
102108
await dialog.getByLabel("Name (optional)").fill(options.stepName);

0 commit comments

Comments
 (0)