test(e2e): make auth.setup.ts idempotent across retries and disable external resources by princejain756 · Pull Request #17818 · Arize-ai/phoenix

princejain756 · 2026-11-09T23:08:26Z

When js/app/tests/auth.setup.ts fails part-way through Attempt 0 against the long-lived in-memory test server, Playwright retries fail because admin@localhost, [email protected], or [email protected] may have already reset their initial passwords or been created on the earlier attempt. In addition, js/app/tests/utils/testServer.mjs left PHOENIX_ALLOW_EXTERNAL_RESOURCES at its default (True) in standard E2E runs, causing page.goto (waitUntil: "load") to block on external Google Fonts and Scarf pixel requests.

This updates auth.setup.ts to inspect the POST /auth/login response status so already-reset accounts log in with their updated password instead of hanging on the password-reset redirect, checks whether [email protected] and [email protected] already exist in /settings/users before opening the Add User dialog, and sets PHOENIX_ALLOW_EXTERNAL_RESOURCES = "False" in the standard E2E branch of testServer.mjs.

…xternal resources Checks /auth/login responses for already-reset passwords, skips re-creating member/viewer accounts if already present in /settings/users, and sets PHOENIX_ALLOW_EXTERNAL_RESOURCES=False in standard e2e runs so page.goto does not wait on external fonts or Scarf telemetry. Fixes Arize-ai#17789

mintlify · 2026-11-09T23:10:19Z

Preview approval required

This pull request comes from a fork, so Mintlify won't build a preview until someone with write access approves commit 475c56b.

Approve preview

arize-phoenix-factory · 2026-11-10T00:12:09Z

The factory recommends ✅ Approve for 59a5f17 and labeled this pull request factory-approved. Read the review · session sesn_01UJZ18uLUmsfYM1UkCsWKP6

cc @RogerHYang

arize-phoenix-factory

✅ Factory review of `59a5f17`: Approve

The diff does what the PR and #17789 ask, every check passes, and there is no blocking finding.

Fit	Checks	Tests	Blocking	Non-blocking	Risk	Depth
✅ Pass	✅ Pass	✅ Not needed	✅ 0	⚠️ 1	⚠️ Low	✅ Glance

Needs attention

⚠️ Non-blocking: Existence check misses users past the first 50 rows js/app/tests/auth.setup.ts:141
⚠️ Risk (Low): Side effects. The standard E2E server now refuses all external resources: fonts, the GitHub star count, the PyPI version check, and the WASM binary download. Sandbox specs still get WASM from the pre-warmed cache or from PHOENIX_WASM_BINARY_PATH in CI.

Details

Intent: Make the Playwright auth bootstrap (js/app/tests/auth.setup.ts) safe to repeat across retries and reuseExistingServer runs against the same in-memory server, where admin, member and viewer may already exist or may have already reset their passwords. Also set PHOENIX_ALLOW_EXTERNAL_RESOURCES=False on the standard E2E server so page.goto doesn't wait on third-party CDNs (#17789).
Fit: Both parts are present.

login() now returns the POST /auth/login response.
A 401 for admin on admin falls back to admin123.
Member and viewer are created only if they aren't already listed.
Their password reset is skipped on a 401.
testServer.mjs sets PHOENIX_ALLOW_EXTERNAL_RESOURCES=False in the non-PXI branch.
Nothing unrelated is changed.
Verified:
oxfmt --check, oxlint (whole app) and tsc --noEmit (js/app) all pass.
I started the E2E test server once and ran playwright test --project=setup against it three times in a row (Chromium). All three passed, and the second and third runs took the new already-set-up paths.
With main's auth.setup.ts against the same server, setup fails with page.waitForURL timing out at auth.setup.ts:63. That is the failure the issue reports, so the change fixes it.
tests/user-management.spec.ts passes on top of the new setup.
The rendered /login HTML has no Google Fonts or Scarf references.
After I added 50 users through createUser, setup failed at line 160, which reproduces the inline finding.
Not verified:
Firefox and WebKit projects, the full E2E suite, and the app-frame and PXI projects.
The sandbox had Node 22 and needed a symlinked Chromium 141 for Playwright 1.64, so the browser runs are close to CI but not identical.
Findings without a diff line: none
Resolved threads: none
Risk: side effects low, performance none, backwards compatibility low, from the area reviewers

Graded against the review criteria.

arize-phoenix-factory · 2026-11-10T00:28:28Z

+    if (
+      !(await page
+        .getByText("[email protected]", { exact: true })
+        .isVisible())
+    ) {
+      await page.getByRole("button", { name: "Add User" }).click();
+      await page.getByLabel("Email").fill("[email protected]");
+      await page.getByLabel("Username").fill("member");
+      await page.getByLabel("Password", { exact: true }).fill("member");
+      await page.getByLabel("Confirm Password").fill("member");
+      await page
+        .getByRole("dialog")
+        .getByLabel("member", { exact: true })
+        .click();
+      await page.getByRole("option", { name: "member" }).click();
+      await page
+        .getByRole("dialog")
+        .getByRole("button", { name: "Add User" })
+        .click();
+      await expect(page.getByTestId("dialog")).not.toBeVisible();
+    }

    // Add viewer user
-    await page.getByRole("button", { name: "Add User" }).click();
-    await page.getByLabel("Email").fill("[email protected]");
-    await page.getByLabel("Username").fill("viewer");
-    await page.getByLabel("Password", { exact: true }).fill("viewer");
-    await page.getByLabel("Confirm Password").fill("viewer");
-    await page
-      .getByRole("dialog")
-      .getByLabel("member", { exact: true })
-      .click();
-    await page.getByRole("option", { name: "viewer" }).click();
-    await page
-      .getByRole("dialog")
-      .getByRole("button", { name: "Add User" })
-      .click();
-    await expect(page.getByTestId("dialog")).not.toBeVisible();
+    if (
+      !(await page
+        .getByText("[email protected]", { exact: true })
+        .isVisible())
+    ) {


Non-blocking: These existence checks only look at the users table's first page. Once more than 50 users sort ahead of [email protected] or [email protected], the setup tries to add a user that already exists and fails. That is the same retry/reuse failure this PR is fixing.

Evidence:

The table is ordered by email (src/phoenix/server/api/queries.py:409-414) and loads 50 rows at a time, with more rows only on scroll (UsersTable.tsx:39,91).

user-management.spec.ts:8,33,66 creates member-<uuid>@…, viewer-<uuid>@… and password-reset-<uuid>@… users on every run and never deletes them. All of them sort before [email protected], because - sorts before @.

With reuseExistingServer: !isCI (playwright.config.ts:139), [email protected] drops off the first page after roughly 16 local runs.

I reproduced this against a long-lived test server:

Setup passed on a second and third run.

I then added 50 more users through createUser, which left [email protected] at row 56.

The next setup run failed at auth.setup.ts:160 with expect(locator).not.toBeVisible(), because the Add User dialog stayed open on the duplicate email.

isVisible() also doesn't wait. It works only because the admin@localhost assertion above has already waited for the query.

CI retries happen before other specs add users, so the CI case the issue reports is fixed. This only affects local reuse.

Fix: Don't use the DOM as the record of which users exist. Pick one of these:

Probe with a login first, for example [email protected] / member123. Treat a success as already set up, and a 401 on both passwords as not created.

Or, when Add User reports that the email already exists, close the dialog and continue.

RogerHYang

Thank you for your contribution!

princejain756 requested a review from a team as a code owner November 9, 2026 23:08

github-project-automation Bot added this to phoenix Nov 9, 2026

github-project-automation Bot moved this to 📘 Todo in phoenix Nov 9, 2026

github-actions Bot added the triage issues that need triage label Nov 9, 2026

RogerHYang added factory-review to be reviewed by the factory and removed triage issues that need triage labels Nov 10, 2026

arize-phoenix-factory Bot reviewed Nov 10, 2026

View reviewed changes

arize-phoenix-factory Bot added factory-approved The factory review recommends approving and removed factory-review to be reviewed by the factory labels Nov 10, 2026

test(e2e): check setup users independently of table pagination

475c56b

RogerHYang approved these changes Nov 10, 2026

View reviewed changes

github-project-automation Bot moved this from 📘 Todo to 👍 Approved in phoenix Nov 10, 2026

RogerHYang enabled auto-merge (squash) November 10, 2026 16:59

RogerHYang merged commit 7fa410b into Arize-ai:main Nov 11, 2026
49 checks passed

github-project-automation Bot moved this from 👍 Approved to ✅ Done in phoenix Nov 11, 2026

This was referenced Nov 13, 2026

chore: sync fork with upstream manabie-com/phoenix#106

Open

chore: sync fork with upstream manabie-com/phoenix#107

Open

Navigation Menu

Repository navigation

test(e2e): make auth.setup.ts idempotent across retries and disable external resources - #17818

test(e2e): make auth.setup.ts idempotent across retries and disable external resources#17818
RogerHYang merged 2 commits into
Arize-ai:mainfrom
princejain756:fix/playwright-auth-setup-idempotent

princejain756 commented Nov 9, 2026

Uh oh!

mintlify Bot commented Nov 9, 2026 •

edited

Loading

Uh oh!

arize-phoenix-factory Bot commented Nov 10, 2026 •

edited

Loading

Uh oh!

arize-phoenix-factory Bot left a comment

Uh oh!

arize-phoenix-factory Bot Nov 10, 2026

Uh oh!

RogerHYang left a comment

Uh oh!

Uh oh!

Reviewers

Uh oh!

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

Conversation

princejain756 commented Nov 9, 2026

Uh oh!

mintlify Bot commented Nov 9, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Preview approval required

Uh oh!

arize-phoenix-factory Bot commented Nov 10, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

arize-phoenix-factory Bot left a comment

Choose a reason for hiding this comment

✅ Factory review of 59a5f17: Approve

Uh oh!

arize-phoenix-factory Bot Nov 10, 2026

Choose a reason for hiding this comment

Uh oh!

RogerHYang left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Reviewers

Uh oh!

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

mintlify Bot commented Nov 9, 2026 •

edited

Loading

arize-phoenix-factory Bot commented Nov 10, 2026 •

edited

Loading

✅ Factory review of `59a5f17`: Approve