fix: server run race by zirain · Pull Request #7964 · envoyproxy/gateway

zirain · 2026-01-15T11:09:57Z

xref: envoyproxy/ai-gateway#1770 (comment) #7880

AIGW required Wait for the runners to close, see #5560.
There's a problem is that runnersDone is blocked until ctx.Done(), which cause goroutine leak when config reloading.

#7880 fix the leak, but there's a race when using WaitGroup with multi goroutinues.

This PR use semaphore with weight 2, to handle all of these cases. see Loader.Wait().

netlify · 2026-01-15T11:10:05Z

✅ Deploy Preview for cerulean-figolla-1f9435 ready!

Name	Link
🔨 Latest commit	`0c735b9`
🔍 Latest deploy log	https://app.netlify.com/projects/cerulean-figolla-1f9435/deploys/697181ff4eac600008cd18e2
😎 Deploy Preview	https://deploy-preview-7964--cerulean-figolla-1f9435.netlify.app
📱 Preview on mobile	Toggle QR Code... Use your smartphone camera to open QR code link.

To edit notification comments on pull requests, go to your Netlify project configuration.

zirain · 2026-01-15T11:21:20Z

failed as excepted:

WARNING: DATA RACE
Write at 0x00c00201e958 by goroutine 129:
  runtime.racewrite()
      <autogenerated>:1 +0x1e
  github.com/envoyproxy/gateway/internal/cmd.server()
      /home/runner/work/gateway/gateway/internal/cmd/server.go:113 +0x491
  github.com/envoyproxy/gateway/internal/cmd.TestServerRun.func1()
      /home/runner/work/gateway/gateway/internal/cmd/server_test.go:43 +0x92

Previous read at 0x00c00201e958 by goroutine 131:
  runtime.raceread()
      <autogenerated>:1 +0x1e
  github.com/envoyproxy/gateway/internal/cmd.server.func1()
      /home/runner/work/gateway/gateway/internal/cmd/server.go:83 +0x84
  github.com/envoyproxy/gateway/internal/envoygateway/config/loader.(*Loader).runHook.func1()
      /home/runner/work/gateway/gateway/internal/envoygateway/config/loader/configloader.go:126 +0xc3
  github.com/envoyproxy/gateway/internal/envoygateway/config/loader.(*Loader).runHook.gowrap1()
      /home/runner/work/gateway/gateway/internal/envoygateway/config/loader/configloader.go:132 +0x4f

Goroutine 129 (running) created at:
  github.com/envoyproxy/gateway/internal/cmd.TestServerRun()
      /home/runner/work/gateway/gateway/internal/cmd/server_test.go:42 +0x147
  testing.tRunner()
      /opt/hostedtoolcache/go/1.25.5/x64/src/testing/testing.go:1934 +0x21c
  testing.(*T).Run.gowrap1()
      /opt/hostedtoolcache/go/1.25.5/x64/src/testing/testing.go:1997 +0x44

Goroutine 131 (running) created at:
  github.com/envoyproxy/gateway/internal/envoygateway/config/loader.(*Loader).runHook()
      /home/runner/work/gateway/gateway/internal/envoygateway/config/loader/configloader.go:124 +0x309
  github.com/envoyproxy/gateway/internal/envoygateway/config/loader.(*Loader).Start()
      /home/runner/work/gateway/gateway/internal/envoygateway/config/loader/configloader.go:46 +0xc4
  github.com/envoyproxy/gateway/internal/cmd.server()
      /home/runner/work/gateway/gateway/internal/cmd/server.go:94 +0x2d7
  github.com/envoyproxy/gateway/internal/cmd.TestServerRun.func1()
      /home/runner/work/gateway/gateway/internal/cmd/server_test.go:43 +0x92

codecov · 2026-01-15T13:16:15Z

Codecov Report

❌ Patch coverage is 66.66667% with 11 lines in your changes missing coverage. Please review.
✅ Project coverage is 73.53%. Comparing base (fc43a31) to head (0c735b9).
⚠️ Report is 4 commits behind head on main.

Files with missing lines	Patch %	Lines
internal/cmd/server.go	57.14%	6 Missing ⚠️
...nternal/envoygateway/config/loader/configloader.go	73.68%	3 Missing and 2 partials ⚠️

Additional details and impacted files

@@            Coverage Diff             @@
##             main    #7964      +/-   ##
==========================================
+ Coverage   72.85%   73.53%   +0.68%     
==========================================
  Files         237      237              
  Lines       35623    35631       +8     
==========================================
+ Hits        25952    26203     +251     
+ Misses       7824     7567     -257     
- Partials     1847     1861      +14

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:

❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.
📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

mathetake · 2026-01-15T23:54:53Z

nice!

arkodg · 2026-01-16T03:30:42Z

can we enable the -race flag before releasing this

zhaohuabing · 2026-01-16T03:32:12Z

Hi @zirain is it easy to hit this?

zirain · 2026-01-16T03:36:48Z

Hi @zirain is it easy to hit this?

the testcase is reproducible.

zirain · 2026-01-16T03:37:51Z

can we enable the -race flag before releasing this

go.test.coverage: go.test.cel ## Run go unit and integration tests in GitHub Actions
	@$(LOG_TARGET)
	KUBEBUILDER_ASSETS="$$($(GO_TOOL) setup-envtest use $(ENVTEST_K8S_VERSION) -p path)" \
		go test ./... --tags=integration -race -coverprofile=coverage.xml -covermode=atomic -coverpkg=./...

it's enabled, the problem is lack of test case.

internal/envoygateway/config/loader/configloader.go

nacx · 2026-01-16T17:11:10Z

We would love to have this merged, as there is an Envoy AI Gateway release waiting for this. @arkodg @jukie mind having another review?

jukie · 2026-01-16T23:01:06Z

internal/cmd/server.go

 				},
 			)
-			return server(cmd.Context(), cmd.OutOrStdout(), cmd.ErrOrStderr(), runnerErrors)
+			started := &atomic.Bool{}


It doesn't look like this is being read or used anywhere, can this be removed?

this's useful to allow us konw when the server is ready and when to trigger a config update.
we can update this to a callback.

Could we structure the tests differently if that's the only place it's being used?

Only a minor nit so non-blocking

arkodg · 2026-01-17T17:32:47Z

this is the 3rd fix in this area, can we take a step back and redesign what server/runner initialization, handling partial failures, and config restarts should look like ?

Signed-off-by: zirain <[email protected]>

zirain · 2026-01-18T01:49:25Z

this is the 3rd fix in this area, can we take a step back and redesign what server/runner initialization, handling partial failures, and config restarts should look like ?

the main reason of the regssion is lack of reason.

the first fix introduce a leak when config reload too much times.
chore: fix goroutine leak #7880 fix the leak but introduce a race condition.

I think the new test case cover them.

Signed-off-by: zirain <[email protected]>

arkodg · 2026-01-18T03:49:51Z

can you elaborate on what the logic is / should be for

dealing with partial runner termination
config reloads that impact runner restarts

zirain · 2026-01-18T04:36:12Z

can you elaborate on what the logic is / should be for

dealing with partial runner termination

config reloads that impact runner restarts

AIGW required Wait for the runners to close, see https://github.com/envoyproxy/gateway/pull/5560/changes.
There's a problem is that runnersDone is blocked until ctx.Done(), which cause goroutine leak when config reloading.

#7880 fix the leak, but there's a race when using WaitGroup with multi goroutinues.

This PR use semaphore with weight 2, to handle all of these cases. see Loader.Wait().

Update above to PR description.

nacx · 2026-01-19T23:51:14Z

this is the 3rd fix in this area, can we take a step back and redesign what server/runner initialization, handling partial failures, and config restarts should look like ?

I agree with the sentiment here, but I also think that given we're close to the desired date for 1.7 release, and the fact that this should be ideally cherry-picked to previous versions, it is convenient to merge the fix (it's safer to cherry-pick a small fix than a refactoring), unblock the projects that depend on this, and then plan and work for a proper redesign.

internal/envoygateway/config/loader/configloader.go

zhaohuabing · 2026-01-20T12:07:35Z

internal/envoygateway/config/loader/configloader.go


-				// TODO: we need to make sure that all runners are stopped, before we start the new ones
-				// Otherwise we might end up with error listening on:8081
-				time.Sleep(3 * time.Second)


Should we check whether the ports used by the goaway runners are available before starting new runners?

we didn't need if we run sequentially.

Signed-off-by: zirain <[email protected]>

internal/envoygateway/config/loader/configloader.go

zhaohuabing

LGTM Thanks!

Signed-off-by: zirain <[email protected]>

zirain · 2026-01-22T06:03:50Z

/retest

* add test Signed-off-by: zirain <[email protected]> * fix race Signed-off-by: zirain <[email protected]> * fix lint Signed-off-by: zirain <[email protected]> * fix Signed-off-by: zirain <[email protected]> * fix Signed-off-by: zirain <[email protected]> * fix lint Signed-off-by: zirain <[email protected]> * use Semaphore instead of WaitGroup Signed-off-by: zirain <[email protected]> * comments Signed-off-by: zirain <[email protected]> * lint Signed-off-by: zirain <[email protected]> * fix Signed-off-by: zirain <[email protected]> * fix lint Signed-off-by: zirain <[email protected]> * callback Signed-off-by: zirain <[email protected]> * fix lint Signed-off-by: zirain <[email protected]> * run hook sequentially Signed-off-by: zirain <[email protected]> * fix lint Signed-off-by: zirain <[email protected]> * rename to cfgMux Signed-off-by: zirain <[email protected]> --------- Signed-off-by: zirain <[email protected]>

* add test Signed-off-by: zirain <[email protected]> * fix race Signed-off-by: zirain <[email protected]> * fix lint Signed-off-by: zirain <[email protected]> * fix Signed-off-by: zirain <[email protected]> * fix Signed-off-by: zirain <[email protected]> * fix lint Signed-off-by: zirain <[email protected]> * use Semaphore instead of WaitGroup Signed-off-by: zirain <[email protected]> * comments Signed-off-by: zirain <[email protected]> * lint Signed-off-by: zirain <[email protected]> * fix Signed-off-by: zirain <[email protected]> * fix lint Signed-off-by: zirain <[email protected]> * callback Signed-off-by: zirain <[email protected]> * fix lint Signed-off-by: zirain <[email protected]> * run hook sequentially Signed-off-by: zirain <[email protected]> * fix lint Signed-off-by: zirain <[email protected]> * rename to cfgMux Signed-off-by: zirain <[email protected]> --------- Signed-off-by: zirain <[email protected]> Signed-off-by: Rudrakh Panigrahi <[email protected]>

* fix: fail fast when unrecoverable discovery errors happens on checking optional CRDs (#7872) * fail fast when unrecoverable discovery errors happens Signed-off-by: Huabing Zhao <[email protected]> * only retry transient errors Signed-off-by: Huabing Zhao <[email protected]> * fix potenial dead lock Signed-off-by: Huabing Zhao <[email protected]> * address comments Signed-off-by: Huabing Zhao <[email protected]> * minor wording Signed-off-by: Huabing Zhao <[email protected]> * create discovery client once Signed-off-by: Huabing Zhao <[email protected]> * fix lint Signed-off-by: Huabing Zhao <[email protected]> * address comments Signed-off-by: Huabing Zhao <[email protected]> * remove redundant logging Signed-off-by: Huabing Zhao <[email protected]> * add e2e test Signed-off-by: Huabing Zhao <[email protected]> * fix test Signed-off-by: Huabing(Robin) Zhao <[email protected]> * fix test Signed-off-by: Huabing(Robin) Zhao <[email protected]> --------- Signed-off-by: Huabing (Robin) Zhao <[email protected]> * fix: extproc is discarded with failOpen is enabled for wasm (#7956) * fix: extproc is discarded with failOpen is enabled for wasm Signed-off-by: Huabing Zhao <[email protected]> * add test Signed-off-by: Huabing (Robin) Zhao <[email protected]> * polish code Signed-off-by: Huabing (Robin) Zhao <[email protected]> * add test Signed-off-by: Huabing (Robin) Zhao <[email protected]> --------- Signed-off-by: Huabing (Robin) Zhao <[email protected]> * fix: sanitize control plane config dump (#7901) * mask secrets Signed-off-by: Huabing Zhao <[email protected]> * address comments Signed-off-by: Huabing (Robin) Zhao <[email protected]> --------- Signed-off-by: Huabing (Robin) Zhao <[email protected]> * fix: server run race (#7964) * add test Signed-off-by: zirain <[email protected]> * fix race Signed-off-by: zirain <[email protected]> * fix lint Signed-off-by: zirain <[email protected]> * fix Signed-off-by: zirain <[email protected]> * fix Signed-off-by: zirain <[email protected]> * fix lint Signed-off-by: zirain <[email protected]> * use Semaphore instead of WaitGroup Signed-off-by: zirain <[email protected]> * comments Signed-off-by: zirain <[email protected]> * lint Signed-off-by: zirain <[email protected]> * fix Signed-off-by: zirain <[email protected]> * fix lint Signed-off-by: zirain <[email protected]> * callback Signed-off-by: zirain <[email protected]> * fix lint Signed-off-by: zirain <[email protected]> * run hook sequentially Signed-off-by: zirain <[email protected]> * fix lint Signed-off-by: zirain <[email protected]> * rename to cfgMux Signed-off-by: zirain <[email protected]> --------- Signed-off-by: zirain <[email protected]> * fix: wrong cluster type with mixed FQDN backend and service backend refs (#7994) * fix: wrong cluster type with mixed FQDN backend and service backend refs Signed-off-by: Huabing (Robin) Zhao <[email protected]> * fix mirror cluster endpoint type Signed-off-by: Huabing (Robin) Zhao <[email protected]> * simplify the test Signed-off-by: Huabing (Robin) Zhao <[email protected]> * update comment Signed-off-by: Huabing (Robin) Zhao <[email protected]> --------- Signed-off-by: Huabing (Robin) Zhao <[email protected]> * fix: merge route match rule with match all route (#8011) Signed-off-by: zirain <[email protected]> * fix gen Signed-off-by: zirain <[email protected]> * fix lint Signed-off-by: zirain <[email protected]> * fix for golang 11.24 Signed-off-by: zirain <[email protected]> * fix lint Signed-off-by: zirain <[email protected]> * fix watch CRD version Signed-off-by: zirain <[email protected]> --------- Signed-off-by: Huabing (Robin) Zhao <[email protected]> Signed-off-by: zirain <[email protected]> Co-authored-by: Huabing (Robin) Zhao <[email protected]>

* fix: extproc is discarded with failOpen is enabled for wasm (#7956) * fix: extproc is discarded with failOpen is enabled for wasm Signed-off-by: Huabing Zhao <[email protected]> * add test Signed-off-by: Huabing (Robin) Zhao <[email protected]> * polish code Signed-off-by: Huabing (Robin) Zhao <[email protected]> * add test Signed-off-by: Huabing (Robin) Zhao <[email protected]> --------- Signed-off-by: Huabing (Robin) Zhao <[email protected]> Signed-off-by: Rudrakh Panigrahi <[email protected]> * fix: sanitize control plane config dump (#7901) * mask secrets Signed-off-by: Huabing Zhao <[email protected]> * address comments Signed-off-by: Huabing (Robin) Zhao <[email protected]> --------- Signed-off-by: Huabing (Robin) Zhao <[email protected]> Signed-off-by: Rudrakh Panigrahi <[email protected]> * fix: server run race (#7964) * add test Signed-off-by: zirain <[email protected]> * fix race Signed-off-by: zirain <[email protected]> * fix lint Signed-off-by: zirain <[email protected]> * fix Signed-off-by: zirain <[email protected]> * fix Signed-off-by: zirain <[email protected]> * fix lint Signed-off-by: zirain <[email protected]> * use Semaphore instead of WaitGroup Signed-off-by: zirain <[email protected]> * comments Signed-off-by: zirain <[email protected]> * lint Signed-off-by: zirain <[email protected]> * fix Signed-off-by: zirain <[email protected]> * fix lint Signed-off-by: zirain <[email protected]> * callback Signed-off-by: zirain <[email protected]> * fix lint Signed-off-by: zirain <[email protected]> * run hook sequentially Signed-off-by: zirain <[email protected]> * fix lint Signed-off-by: zirain <[email protected]> * rename to cfgMux Signed-off-by: zirain <[email protected]> --------- Signed-off-by: zirain <[email protected]> Signed-off-by: Rudrakh Panigrahi <[email protected]> * fix: wrong cluster type with mixed FQDN backend and service backend refs (#7994) * fix: wrong cluster type with mixed FQDN backend and service backend refs Signed-off-by: Huabing (Robin) Zhao <[email protected]> * fix mirror cluster endpoint type Signed-off-by: Huabing (Robin) Zhao <[email protected]> * simplify the test Signed-off-by: Huabing (Robin) Zhao <[email protected]> * update comment Signed-off-by: Huabing (Robin) Zhao <[email protected]> --------- Signed-off-by: Huabing (Robin) Zhao <[email protected]> Signed-off-by: Rudrakh Panigrahi <[email protected]> * fix: fail fast when unrecoverable discovery errors happens on checking optional CRDs (#7872) * fail fast when unrecoverable discovery errors happens Signed-off-by: Huabing Zhao <[email protected]> * only retry transient errors Signed-off-by: Huabing Zhao <[email protected]> * fix potenial dead lock Signed-off-by: Huabing Zhao <[email protected]> * address comments Signed-off-by: Huabing Zhao <[email protected]> * minor wording Signed-off-by: Huabing Zhao <[email protected]> * create discovery client once Signed-off-by: Huabing Zhao <[email protected]> * fix lint Signed-off-by: Huabing Zhao <[email protected]> * address comments Signed-off-by: Huabing Zhao <[email protected]> * remove redundant logging Signed-off-by: Huabing Zhao <[email protected]> * add e2e test Signed-off-by: Huabing Zhao <[email protected]> * fix test Signed-off-by: Huabing(Robin) Zhao <[email protected]> * fix test Signed-off-by: Huabing(Robin) Zhao <[email protected]> --------- Signed-off-by: Huabing (Robin) Zhao <[email protected]> Signed-off-by: Rudrakh Panigrahi <[email protected]> * fix: merge route match rule with match all route (#8011) Signed-off-by: zirain <[email protected]> Signed-off-by: Rudrakh Panigrahi <[email protected]> * fix: do not set autoHTTPConfig when used mixed(HTTP + HTTPS) backends (#7950) * fix: do not set autoHTTPConfig when used mixed backend Signed-off-by: zirain <[email protected]> * release notes Signed-off-by: zirain <[email protected]> * fix Signed-off-by: zirain <[email protected]> * add e2e Signed-off-by: zirain <[email protected]> --------- Signed-off-by: zirain <[email protected]> Signed-off-by: Rudrakh Panigrahi <[email protected]> * fix: backend tls default namespace (#7987) Signed-off-by: Huabing (Robin) Zhao <[email protected]> Signed-off-by: Rudrakh Panigrahi <[email protected]> * fix: race in gatewaapi runner (#8037) * add testcase Signed-off-by: zirain <[email protected]> * fix Signed-off-by: zirain <[email protected]> * simply Signed-off-by: zirain <[email protected]> --------- Signed-off-by: zirain <[email protected]> Signed-off-by: Rudrakh Panigrahi <[email protected]> * [release/v1.6] v1.6.3 release notes (#8054) Signed-off-by: Rudrakh Panigrahi <[email protected]> * v1.6.3 version Signed-off-by: Rudrakh Panigrahi <[email protected]> * fix gen-check Signed-off-by: Rudrakh Panigrahi <[email protected]> * fix lint Signed-off-by: Rudrakh Panigrahi <[email protected]> --------- Signed-off-by: Huabing (Robin) Zhao <[email protected]> Signed-off-by: Rudrakh Panigrahi <[email protected]> Signed-off-by: zirain <[email protected]> Co-authored-by: Huabing (Robin) Zhao <[email protected]> Co-authored-by: zirain <[email protected]>

* add test Signed-off-by: zirain <[email protected]> * fix race Signed-off-by: zirain <[email protected]> * fix lint Signed-off-by: zirain <[email protected]> * fix Signed-off-by: zirain <[email protected]> * fix Signed-off-by: zirain <[email protected]> * fix lint Signed-off-by: zirain <[email protected]> * use Semaphore instead of WaitGroup Signed-off-by: zirain <[email protected]> * comments Signed-off-by: zirain <[email protected]> * lint Signed-off-by: zirain <[email protected]> * fix Signed-off-by: zirain <[email protected]> * fix lint Signed-off-by: zirain <[email protected]> * callback Signed-off-by: zirain <[email protected]> * fix lint Signed-off-by: zirain <[email protected]> * run hook sequentially Signed-off-by: zirain <[email protected]> * fix lint Signed-off-by: zirain <[email protected]> * rename to cfgMux Signed-off-by: zirain <[email protected]> --------- Signed-off-by: zirain <[email protected]> Signed-off-by: Sadmi Bouhafs <[email protected]>

zirain requested a review from a team as a code owner January 15, 2026 11:09

This was referenced Jan 15, 2026

chore: fix goroutine leak #7880

Merged

fix: fail fast when unrecoverable discovery errors happens on checking optional CRDs #7872

Merged

zirain added cherrypick/release-v1.6.3 cherrypick/release-v1.5.8 labels Jan 15, 2026

jukie previously approved these changes Jan 16, 2026

View reviewed changes

zhaohuabing reviewed Jan 16, 2026

View reviewed changes

internal/envoygateway/config/loader/configloader.go Outdated Show resolved Hide resolved

zirain dismissed jukie’s stale review via 47713f2 January 16, 2026 06:18

zirain force-pushed the fix/waitgroup-race branch 2 times, most recently from 9699599 to 6483c24 Compare January 16, 2026 07:56

zirain changed the title ~~fix: WaitGroup race~~ fix: server run race Jan 16, 2026

jukie reviewed Jan 16, 2026

View reviewed changes

zirain force-pushed the fix/waitgroup-race branch from a5fad1d to de06900 Compare January 17, 2026 10:36

zirain added 7 commits January 18, 2026 09:42

add test

39b77b4

Signed-off-by: zirain <[email protected]>

fix race

7d1a6d3

Signed-off-by: zirain <[email protected]>

fix lint

6a572ea

Signed-off-by: zirain <[email protected]>

fix

a14d603

Signed-off-by: zirain <[email protected]>

fix

d69b914

Signed-off-by: zirain <[email protected]>

fix lint

017afb8

Signed-off-by: zirain <[email protected]>

use Semaphore instead of WaitGroup

b48915d

Signed-off-by: zirain <[email protected]>

fix lint

976bc3f

Signed-off-by: zirain <[email protected]>

zhaohuabing reviewed Jan 20, 2026

View reviewed changes

internal/envoygateway/config/loader/configloader.go Outdated Show resolved Hide resolved

zhaohuabing reviewed Jan 20, 2026

View reviewed changes

zirain added 2 commits January 20, 2026 20:29

run hook sequentially

2a0aef7

Signed-off-by: zirain <[email protected]>

fix lint

e5af0e3

Signed-off-by: zirain <[email protected]>

zhaohuabing reviewed Jan 22, 2026

View reviewed changes

internal/envoygateway/config/loader/configloader.go Outdated Show resolved Hide resolved

zhaohuabing previously approved these changes Jan 22, 2026

View reviewed changes

zhaohuabing requested a review from arkodg January 22, 2026 01:24

zirain added 2 commits January 22, 2026 09:45

rename to cfgMux

102237c

Signed-off-by: zirain <[email protected]>

Merge branch 'main' into fix/waitgroup-race

0c735b9

zirain dismissed zhaohuabing’s stale review via 0c735b9 January 22, 2026 01:48

zirain requested review from jukie and zhaohuabing January 22, 2026 01:48

zhaohuabing approved these changes Jan 22, 2026

View reviewed changes

arkodg requested review from a team January 22, 2026 05:38

arkodg approved these changes Jan 22, 2026

View reviewed changes

zirain merged commit 338ebea into envoyproxy:main Jan 22, 2026
141 of 151 checks passed

zirain deleted the fix/waitgroup-race branch January 22, 2026 08:18

zirain mentioned this pull request Jan 24, 2026

Data race while running tests that fixes with retry #8035

Closed

Conversation

zirain commented Jan 15, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

netlify bot commented Jan 15, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

✅ Deploy Preview for cerulean-figolla-1f9435 ready!

Uh oh!

zirain commented Jan 15, 2026

Uh oh!

codecov bot commented Jan 15, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Codecov Report

Uh oh!

mathetake commented Jan 15, 2026

Uh oh!

arkodg commented Jan 16, 2026

Uh oh!

zhaohuabing commented Jan 16, 2026

Uh oh!

zirain commented Jan 16, 2026

Uh oh!

zirain commented Jan 16, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Uh oh!

nacx commented Jan 16, 2026

Uh oh!

jukie Jan 16, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

zirain Jan 17, 2026

Choose a reason for hiding this comment

Uh oh!

jukie Jan 17, 2026

Choose a reason for hiding this comment

Uh oh!

jukie Jan 17, 2026

Choose a reason for hiding this comment

Uh oh!

arkodg commented Jan 17, 2026

Uh oh!

zirain commented Jan 18, 2026

Uh oh!

arkodg commented Jan 18, 2026

Uh oh!

zirain commented Jan 18, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

nacx commented Jan 19, 2026

Uh oh!

Uh oh!

zhaohuabing Jan 20, 2026

Choose a reason for hiding this comment

Uh oh!

zirain Jan 20, 2026

Choose a reason for hiding this comment

Uh oh!

Uh oh!

zhaohuabing left a comment

Choose a reason for hiding this comment

Uh oh!

zirain commented Jan 22, 2026

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

6 participants

zirain commented Jan 15, 2026 •

edited

Loading

netlify bot commented Jan 15, 2026 •

edited

Loading

codecov bot commented Jan 15, 2026 •

edited

Loading

zirain commented Jan 16, 2026 •

edited

Loading

jukie Jan 16, 2026 •

edited

Loading

zirain commented Jan 18, 2026 •

edited

Loading