Skip to content

test(typed-routes): fix flaky tests by increasing retry timeout#91923

Merged
sokra merged 1 commit intocanaryfrom
sokra/flaky-test-9
Mar 26, 2026
Merged

test(typed-routes): fix flaky tests by increasing retry timeout#91923
sokra merged 1 commit intocanaryfrom
sokra/flaky-test-9

Conversation

@sokra
Copy link
Copy Markdown
Member

@sokra sokra commented Mar 26, 2026

Fixing a bug

What?

Three tests in test/e2e/app-dir/typed-routes/typed-routes.test.ts were intermittently failing with Failed to retry within 3000ms:

  • typed-routes › should generate route types correctly
  • typed-routes › should have passing tsc after start
  • typed-routes › should correctly convert custom route patterns from path-to-regexp to bracket syntax

Why?

The test harness marks the dev server as "ready" when it sees ✓ Ready in X in stdout. However, the server logs this message before completing the full initialization sequence in start-server.ts — specifically before getRequestHandlers() resolves, which chains into setupDevBundler(). Route type generation (routes.d.ts) only completes inside setupDevBundler's Watchpack aggregated event handler.

The gap between the "Ready" log and actual routes.d.ts completion can exceed 3000ms on loaded CI runners, causing the tests' retry() calls to time out before the file exists.

A secondary issue in should have passing tsc after start: the test called next.stop() immediately after the server was considered ready, before routes.d.ts was written. This meant tsc ran against a project missing the generated types, causing type errors even though the server had been healthy.

How?

  • Increased the retry() timeout from the default 3000ms to 30000ms for the three affected tests.
  • Added a retry() guard in should have passing tsc after start to wait for routes.d.ts to be present and correct before stopping the server. This ensures tsc sees complete generated types.

Verified by running the full test file 3 consecutive times locally with NEXT_SKIP_ISOLATE=1 NEXT_TEST_MODE=dev pnpm testheadless — all passes, all 5 tests green each run.

The dev server emits "Ready in X" before `setupDevBundler` completes,
so `routes.d.ts` may not exist yet when tests start polling. The
default 3000ms retry window is too short on loaded CI runners.

Increase the timeout to 30000ms for the three affected tests, and add
a guard in `should have passing tsc after start` to ensure
`routes.d.ts` is written before stopping the server and running tsc.

Co-Authored-By: Claude <[email protected]>
@nextjs-bot nextjs-bot added created-by: Turbopack team PRs by the Turbopack team. tests labels Mar 26, 2026
@sokra sokra marked this pull request as ready for review March 26, 2026 00:23
@sokra sokra requested a review from eps1lon March 26, 2026 00:23
@nextjs-bot
Copy link
Copy Markdown
Collaborator

Stats from current PR

✅ No significant changes detected

📊 All Metrics
📖 Metrics Glossary

Dev Server Metrics:

  • Listen = TCP port starts accepting connections
  • First Request = HTTP server returns successful response
  • Cold = Fresh build (no cache)
  • Warm = With cached build artifacts

Build Metrics:

  • Fresh = Clean build (no .next directory)
  • Cached = With existing .next directory

Change Thresholds:

  • Time: Changes < 50ms AND < 10%, OR < 2% are insignificant
  • Size: Changes < 1KB AND < 1% are insignificant
  • All other changes are flagged to catch regressions

⚡ Dev Server

Metric Canary PR Change Trend
Cold (Listen) 455ms 456ms ▄█▁▁▁
Cold (Ready in log) 439ms 438ms ▄█▁▁▁
Cold (First Request) 1.130s 1.101s ▃▅▂▂▂
Warm (Listen) 457ms 456ms ▅█▁▁▁
Warm (Ready in log) 443ms 442ms ▅█▁▁▁
Warm (First Request) 346ms 344ms ▃▇▁▁▁
📦 Dev Server (Webpack) (Legacy)

📦 Dev Server (Webpack)

Metric Canary PR Change Trend
Cold (Listen) 455ms 456ms ▁▁▁▁▁
Cold (Ready in log) 445ms 443ms ▁▂▃▁▁
Cold (First Request) 1.948s 1.947s ▄▄█▁▄
Warm (Listen) 457ms 456ms ▁▁▁▁▁
Warm (Ready in log) 442ms 444ms ▁▂▃▁▁
Warm (First Request) 1.953s 1.954s ▃▄▂▁▃

⚡ Production Builds

Metric Canary PR Change Trend
Fresh Build 3.737s 3.819s ▃▆▁▁▁
Cached Build 3.793s 3.768s ▃▆▁▁▁
📦 Production Builds (Webpack) (Legacy)

📦 Production Builds (Webpack)

Metric Canary PR Change Trend
Fresh Build 14.527s 14.729s ▁▂▄▁▂
Cached Build 14.630s 14.664s ▂▂▄▁▁
node_modules Size 484 MB 484 MB ▁▁▁▁▁
📦 Bundle Sizes

Bundle Sizes

⚡ Turbopack

Client

Main Bundles
Canary PR Change
004mk2--g8dqn.js gzip 70.8 kB N/A -
00ylg--xxmfi9.js gzip 155 B N/A -
02fkg8wfh0iju.js gzip 9.19 kB N/A -
04a7d-r19peqj.js gzip 160 B N/A -
050zwt5xh_0tx.js gzip 10.4 kB N/A -
06rvbj82bhyo0.js gzip 13 kB N/A -
087fzjd-gvlzv.js gzip 450 B N/A -
0cz1d0mv5g_q7.js gzip 39.4 kB 39.4 kB
0dl72q89_7i_j.js gzip 156 B N/A -
0ppxcl_z43mad.js gzip 8.52 kB N/A -
0s-akjt_hclkj.js gzip 65.7 kB N/A -
12_kq9cdklvl5.js gzip 156 B N/A -
19oha6-znmkcv.js gzip 8.55 kB N/A -
1elt1qium-r2m.css gzip 115 B 115 B
1fj4_ea_e2gsp.js gzip 154 B N/A -
1jjguptdz5lo8.js gzip 154 B N/A -
1q7d5eqv50ngw.js gzip 161 B N/A -
1xzxl0s9j2n81.js gzip 48.9 kB N/A -
1yl5jy653a49k.js gzip 13.7 kB N/A -
1zbfxdhh1rhg-.js gzip 156 B N/A -
1zi18gdtym70f.js gzip 7.61 kB N/A -
26elcgxnn9zjd.js gzip 8.52 kB N/A -
2900hudr6gvm0.js gzip 2.28 kB N/A -
2ihlsvbo2c9e8.js gzip 215 B 215 B
2lv2js3kmdeho.js gzip 8.48 kB N/A -
2rehygrd36hqv.js gzip 8.58 kB N/A -
2srwswih0m9_h.js gzip 13.3 kB N/A -
3-p9p9mheqhzx.js gzip 8.55 kB N/A -
31030bryqpolg.js gzip 8.53 kB N/A -
31dx5nmrzzuy7.js gzip 225 B N/A -
3dezu3y1nsgbq.js gzip 153 B N/A -
3fhu7dif32jhi.js gzip 158 B N/A -
3iu80eefg23ae.js gzip 9.77 kB N/A -
3k-48b78ys_vy.js gzip 10.1 kB N/A -
3lcqt7dptotub.js gzip 168 B N/A -
3m7-5rfj0avoz.js gzip 12.9 kB N/A -
3qbtuir3hwnml.js gzip 155 B N/A -
3s1kzqdfs296w.js gzip 157 B N/A -
3uqce_6sa526g.js gzip 8.47 kB N/A -
3yurjqk-sjs3y.js gzip 1.46 kB N/A -
421vzwdt9j1b_.js gzip 5.62 kB N/A -
turbopack-05..dw6l.js gzip 4.16 kB N/A -
turbopack-1_..e8xd.js gzip 4.14 kB N/A -
turbopack-1b..535p.js gzip 4.16 kB N/A -
turbopack-1c..u91g.js gzip 4.16 kB N/A -
turbopack-1d..be4p.js gzip 4.16 kB N/A -
turbopack-1h..7d3s.js gzip 4.16 kB N/A -
turbopack-1l..t6hi.js gzip 4.17 kB N/A -
turbopack-1m..q0f0.js gzip 4.16 kB N/A -
turbopack-1y..h6uu.js gzip 4.16 kB N/A -
turbopack-21..2171.js gzip 4.16 kB N/A -
turbopack-23..xqh2.js gzip 4.16 kB N/A -
turbopack-2t..k165.js gzip 4.15 kB N/A -
turbopack-2z..e9qi.js gzip 4.16 kB N/A -
turbopack-3c..3d3a.js gzip 4.16 kB N/A -
03dgzoo-qf3sm.js gzip N/A 9.19 kB -
05tx5f25dlivn.js gzip N/A 8.53 kB -
0c7ez6p2qc57f.js gzip N/A 5.62 kB -
0ce2-0udgb1xg.js gzip N/A 156 B -
0n95_eeqgrhcd.js gzip N/A 13.7 kB -
0qnwuk92m8i7o.js gzip N/A 10.4 kB -
0r4wrn6n0ue2m.js gzip N/A 8.55 kB -
0rp0fodtbt_6m.js gzip N/A 8.52 kB -
0sfck-km4dl1k.js gzip N/A 8.47 kB -
0x0xuhmxzwkp8.js gzip N/A 8.47 kB -
1_t_23txaa4w1.js gzip N/A 154 B -
1-wdvgxnzicj7.js gzip N/A 1.46 kB -
11u6nxujb2eg4.js gzip N/A 450 B -
15_qn1ql5_cbl.js gzip N/A 48.8 kB -
15v0gax1fapgn.js gzip N/A 170 B -
1hiwsniqd4wpt.js gzip N/A 164 B -
1rgr3b6lyp0ry.js gzip N/A 156 B -
1yber4lya1-wo.js gzip N/A 7.6 kB -
2k9ax08cjl2id.js gzip N/A 12.9 kB -
2lms6k76q5-6m.js gzip N/A 13.3 kB -
2p2c6xj66y9b3.js gzip N/A 157 B -
2q3hz-jtbjs52.js gzip N/A 9.77 kB -
2qx4twi9i3xus.js gzip N/A 2.28 kB -
2srnqic6tvxxd.js gzip N/A 8.52 kB -
2x__oo4_6blmr.js gzip N/A 157 B -
30l7m4nayp73a.js gzip N/A 8.55 kB -
38rr7d3kfutni.js gzip N/A 13 kB -
39p0jmvjhf5s1.js gzip N/A 156 B -
3h_ecpiaatwgc.js gzip N/A 10.1 kB -
3h3-5cuk8apun.js gzip N/A 159 B -
3i4rmegtgcun5.js gzip N/A 152 B -
3ity0aahajapd.js gzip N/A 225 B -
3jl5zguc6kwdc.js gzip N/A 158 B -
3tiezn7-s-9x6.js gzip N/A 156 B -
3tm-7k-_1hn2g.js gzip N/A 70.8 kB -
3yhd7_8g4zcd9.js gzip N/A 157 B -
43mlw9dy_8f02.js gzip N/A 8.58 kB -
44ak7l-ozlmhf.js gzip N/A 65.7 kB -
turbopack-00..ci22.js gzip N/A 4.16 kB -
turbopack-05..98n1.js gzip N/A 4.16 kB -
turbopack-08..lccf.js gzip N/A 4.16 kB -
turbopack-0g..hv43.js gzip N/A 4.16 kB -
turbopack-0j..aj6r.js gzip N/A 4.16 kB -
turbopack-0l..xgp5.js gzip N/A 4.16 kB -
turbopack-0o..c8zk.js gzip N/A 4.18 kB -
turbopack-0p..edl5.js gzip N/A 4.16 kB -
turbopack-1_..d5fb.js gzip N/A 4.16 kB -
turbopack-1i..abxy.js gzip N/A 4.16 kB -
turbopack-2n..exfi.js gzip N/A 4.16 kB -
turbopack-31..znx_.js gzip N/A 4.16 kB -
turbopack-3e..2lsc.js gzip N/A 4.16 kB -
turbopack-3y..kfe3.js gzip N/A 4.13 kB -
Total 463 kB 463 kB ⚠️ +7 B

Server

Middleware
Canary PR Change
middleware-b..fest.js gzip 719 B 715 B
Total 719 B 715 B ✅ -4 B
Build Details
Build Manifests
Canary PR Change
_buildManifest.js gzip 430 B 434 B
Total 430 B 434 B ⚠️ +4 B

📦 Webpack

Client

Main Bundles
Canary PR Change
5528-HASH.js gzip 5.54 kB N/A -
6280-HASH.js gzip 60.6 kB N/A -
6335.HASH.js gzip 169 B N/A -
912-HASH.js gzip 4.59 kB N/A -
e8aec2e4-HASH.js gzip 62.7 kB N/A -
framework-HASH.js gzip 59.7 kB 59.7 kB
main-app-HASH.js gzip 256 B 254 B
main-HASH.js gzip 39.3 kB 39.2 kB
webpack-HASH.js gzip 1.68 kB 1.68 kB
262-HASH.js gzip N/A 4.59 kB -
2889.HASH.js gzip N/A 169 B -
5602-HASH.js gzip N/A 5.55 kB -
6948ada0-HASH.js gzip N/A 62.7 kB -
9544-HASH.js gzip N/A 61.3 kB -
Total 235 kB 235 kB ⚠️ +674 B
Polyfills
Canary PR Change
polyfills-HASH.js gzip 39.4 kB 39.4 kB
Total 39.4 kB 39.4 kB
Pages
Canary PR Change
_app-HASH.js gzip 194 B 194 B
_error-HASH.js gzip 183 B 180 B 🟢 3 B (-2%)
css-HASH.js gzip 331 B 330 B
dynamic-HASH.js gzip 1.81 kB 1.81 kB
edge-ssr-HASH.js gzip 256 B 256 B
head-HASH.js gzip 351 B 352 B
hooks-HASH.js gzip 384 B 383 B
image-HASH.js gzip 580 B 581 B
index-HASH.js gzip 260 B 260 B
link-HASH.js gzip 2.51 kB 2.51 kB
routerDirect..HASH.js gzip 320 B 319 B
script-HASH.js gzip 386 B 386 B
withRouter-HASH.js gzip 315 B 315 B
1afbb74e6ecf..834.css gzip 106 B 106 B
Total 7.98 kB 7.98 kB ✅ -1 B

Server

Edge SSR
Canary PR Change
edge-ssr.js gzip 125 kB 125 kB
page.js gzip 270 kB 270 kB
Total 395 kB 395 kB ⚠️ +228 B
Middleware
Canary PR Change
middleware-b..fest.js gzip 615 B 614 B
middleware-r..fest.js gzip 156 B 155 B
middleware.js gzip 44.1 kB 44.1 kB
edge-runtime..pack.js gzip 842 B 842 B
Total 45.7 kB 45.7 kB ✅ -33 B
Build Details
Build Manifests
Canary PR Change
_buildManifest.js gzip 715 B 718 B
Total 715 B 718 B ⚠️ +3 B
Build Cache
Canary PR Change
0.pack gzip 4.33 MB 4.33 MB
index.pack gzip 110 kB 110 kB
index.pack.old gzip 109 kB 110 kB
Total 4.55 MB 4.55 MB ✅ -1.3 kB

🔄 Shared (bundler-independent)

Runtimes
Canary PR Change
app-page-exp...dev.js gzip 334 kB 334 kB
app-page-exp..prod.js gzip 182 kB 182 kB
app-page-tur...dev.js gzip 334 kB 334 kB
app-page-tur..prod.js gzip 181 kB 181 kB
app-page-tur...dev.js gzip 330 kB 330 kB
app-page-tur..prod.js gzip 180 kB 180 kB
app-page.run...dev.js gzip 331 kB 331 kB
app-page.run..prod.js gzip 180 kB 180 kB
app-route-ex...dev.js gzip 76.2 kB 76.2 kB
app-route-ex..prod.js gzip 51.9 kB 51.9 kB
app-route-tu...dev.js gzip 76.3 kB 76.3 kB
app-route-tu..prod.js gzip 51.9 kB 51.9 kB
app-route-tu...dev.js gzip 75.9 kB 75.9 kB
app-route-tu..prod.js gzip 51.6 kB 51.6 kB
app-route.ru...dev.js gzip 75.8 kB 75.8 kB
app-route.ru..prod.js gzip 51.6 kB 51.6 kB
dist_client_...dev.js gzip 324 B 324 B
dist_client_...dev.js gzip 326 B 326 B
dist_client_...dev.js gzip 318 B 318 B
dist_client_...dev.js gzip 317 B 317 B
pages-api-tu...dev.js gzip 43.5 kB 43.5 kB
pages-api-tu..prod.js gzip 33.1 kB 33.1 kB
pages-api.ru...dev.js gzip 43.4 kB 43.4 kB
pages-api.ru..prod.js gzip 33.1 kB 33.1 kB
pages-turbo....dev.js gzip 52.9 kB 52.9 kB
pages-turbo...prod.js gzip 38.7 kB 38.7 kB
pages.runtim...dev.js gzip 52.8 kB 52.8 kB
pages.runtim..prod.js gzip 38.7 kB 38.7 kB
server.runti..prod.js gzip 62.5 kB 62.5 kB
Total 2.96 MB 2.96 MB
📎 Tarball URL
https://vercel-packages.vercel.app/next/commits/38571f8c121ec28b3c7aa0b75ce5c68459e2fdaf/next

@nextjs-bot
Copy link
Copy Markdown
Collaborator

nextjs-bot commented Mar 26, 2026

Failing test suites

Commit: 6366b7d | About building and testing Next.js

pnpm test-start test/production/app-dir/server-action-period-hash/server-action-period-hash-custom-key.test.ts (job)

  • app-dir - server-action-period-hash-custom-key > should have a different manifest if the encryption key from process env is changed (DD)
  • app-dir - server-action-period-hash-custom-key > should have the same manifest if the encryption key from process env is same (DD)
Expand output

● app-dir - server-action-period-hash-custom-key › should have a different manifest if the encryption key from process env is changed

can not run export while server is running, use next.stop() first

  251 |   ) {
  252 |     if (this.childProcess) {
> 253 |       throw new Error(
      |             ^
  254 |         `can not run export while server is running, use next.stop() first`
  255 |       )
  256 |     }

  at NextStartInstance.build (lib/next-modes/next-start.ts:253:13)
  at Object.build (production/app-dir/server-action-period-hash/server-action-period-hash-custom-key.test.ts:18:16)

● app-dir - server-action-period-hash-custom-key › should have the same manifest if the encryption key from process env is same

can not run export while server is running, use next.stop() first

  251 |   ) {
  252 |     if (this.childProcess) {
> 253 |       throw new Error(
      |             ^
  254 |         `can not run export while server is running, use next.stop() first`
  255 |       )
  256 |     }

  at NextStartInstance.build (lib/next-modes/next-start.ts:253:13)
  at Object.build (production/app-dir/server-action-period-hash/server-action-period-hash-custom-key.test.ts:32:16)

pnpm test-start test/production/app-dir/server-action-period-hash/server-action-period-hash.test.ts (job)

  • segment cache (revalidation) > re-fetch visible links after a navigation, if needed (DD)
Expand output

● segment cache (revalidation) › re-fetch visible links after a navigation, if needed

thrown: "Exceeded timeout of 60000 ms for a test.
Add a timeout value to this test to increase the timeout, if this is a long-running test. See https://jestjs.io/docs/api#testname-fn-timeout."

  50 |       }
  51 |
> 52 |       const result = Reflect.apply(target, thisArg, args)
     |                              ^
  53 |       return typeof result === 'function' ? wrapJestTestFn(result) : result
  54 |     },
  55 |     get(target, prop, receiver) {

  at Object.apply (lib/e2e-utils/index.ts:52:30)
  at it (e2e/app-dir/segment-cache/revalidation/segment-cache-revalidation.test.ts:291:3)
  at Object.describe (e2e/app-dir/segment-cache/revalidation/segment-cache-revalidation.test.ts:8:1)

@sokra sokra requested a review from timneutkens March 26, 2026 09:53
@sokra sokra merged commit f7d4870 into canary Mar 26, 2026
152 of 156 checks passed
@sokra sokra deleted the sokra/flaky-test-9 branch March 26, 2026 21:20
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

created-by: Turbopack team PRs by the Turbopack team. tests

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants