Skip to content

fix(admin): coordinate initial URI and discovery configuration registration - #7888

Open
cloudbase-ai wants to merge 2 commits into
apache:masterfrom
cloudbase-ai:fix/issue-5839-discovery-registration
Open

cloudbase-ai wants to merge 2 commits into
apache:masterfrom
cloudbase-ai:fix/issue-5839-discovery-registration

Conversation

@cloudbase-ai

Copy link
Copy Markdown
Contributor

URI registration can create an automatic LOCAL discovery before the client's Zookeeper/Nacos configuration arrives. Admin then reuses the LOCAL handler without rebinding it, so the configured discovery never owns the selector. Concurrent URI writers can also create duplicate handler/relation rows.

Fixes #5839.

  • Serialize the complete URI registration and its retries with the same selector row lock used by configured discovery. Keep a consistent selector→plugin lock order for shared remote discovery creation; unrelated selectors do not hold a shared plugin lock during URI health checks. Use explicit current reads for handler/discovery state under MySQL repeatable-read transactions.
  • Promote only the automatically generated LOCAL fallback: preserve the handler/relation identity, apply configured properties/listener, clear the old LOCAL upstreams and committed runtime cache even for an empty remote registry, and remove the fallback discovery when unreferenced. Preserve established remote/manual bindings, and skip local URI handle/upstream mutations once remote discovery owns the selector.
  • Publish the URI discovery event after commit. Use a transaction template around the actual implementation so the existing fallback/retry self-invocation path has the same atomic behavior.

Validation:

  • Two regression tests failed before the fix: configured discovery did not rebind the LOCAL handler, and local registration returned a remote handler.
  • JDK 17 Admin/dependency verify: 2,865 tests, 0 failures, 0 errors, 6 skipped; Checkstyle and RAT passed.
  • Real ZooKeeper integration suite: 7 cases passed on H2, MySQL 8.0 ARM64 and PostgreSQL 15 amd64. Each native run initialized the actual shipped schema and used the real Spring/MyBatis services/drivers. Cases cover URI-first/config-first arrival, concurrent writers, rollback, independent selectors under a held selector lock, empty-registry takeover, and the existing Admin HTTP registration/registry-watch/WebSocket flow.
  • Unit coverage also verifies existing remote bindings, namespace rejection and commit-only publication. Independent final diff review found no definite functional defect.

The registration tests disable independent periodic HTTP health checking to isolate registration ordering; #7881 tracks that separate health-cache path. No two-Admin deployment or full Bootstrap request E2E is claimed. Existing duplicate relations are not automatically merged. No schema migration or dependency change is required; Oracle/OceanBase/openGauss registration integration was not executed locally.

Aias00
Aias00 previously approved these changes Oct 21, 2026

@Aias00 Aias00 left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Review: coordinate initial URI and configured discovery registration

This is a solid fix, and I checked the mechanism against the schema and the surrounding code rather than the description alone.

What I verified

  • The takeover logic is correct. For a selector whose handler still points at the automatic LOCAL fallback, bindingDiscovery repoints the existing handler (updateSelective) to the configured discovery, keeps the discovery_rel row untouched, deletes the discovery_upstream rows by handler id, and only then removes the unreferenced fallback discovery row. Relation/handler identity is genuinely preserved, so nothing on the gateway side sees a new selector binding.
  • The empty-registry case is covered. Clearing runtime state cannot rely on fetchAll alone: fetchAll no-ops when discoveryServiceCache has no entry for the discovery id, so removeSelectorUpstream is what guarantees the DELETE event fires. Calling it before createDiscovery is the right order — the DELETE lands, then the remote snapshot overwrites it.
  • Lock order is consistent. registerDiscoveryConfig locks selector → plugin; registerDefaultDiscovery and registerURIWithLock lock only the selector. Nothing in the new code takes the plugin lock before a selector lock, and SelectorService#findByNameAndPluginNameAndNamespaceIdForUpdate (plugin lock first) is no longer on either path.
  • The lock footprint is actually narrow, which is the part I expected to be a problem. selector has idx_selector_ns_plugin_name (namespace_id, plugin_id, selector_name), so findByNameAndPluginNameAndNamespaceIdForUpdate next-key-locks a single record instead of scanning. selectBySelectorIdForUpdate resolves through idx_discovery_rel_selector and joins discovery_handler by primary key, so it locks one rel row plus one handler row. selectByPluginNameAndLevelAndNamespaceIdAndTypeForUpdate uses idx_discovery_ns_plugin. This matters because FOR UPDATE on an unindexed predicate would have serialized every registration in the cluster — it does not.
  • registerDefaultDiscovery returning null is handled at the only production call site. registerURIWithLock returns early before buildHandle/updateSelective, so no local upstream or handle mutation happens once a remote discovery owns the selector. The ShenyuException it now throws on namespace/plugin mismatch cannot fire on the URI path, because the selector is resolved by plugin name in the same statement.
  • Transaction and event ordering hold: nativeCreateOrUpdate is a plain DB write with no event publication, updateSelective goes through the dispatcher's afterCommit, and doDiscoveryLocal now registers its publish after commit, so nothing is delivered for a rolled-back registration. testRolledBackUriRegistrationLeavesNoDiscoveryBinding pins exactly that.

The tests are unusually good for this area: arrival-order parameterization, two concurrent writers, an empty registry, rollback, and the independence assertion with a held selector lock all target real failure modes rather than coverage.

Non-blocking comments

  1. doDiscoveryLocal no longer tolerates a null handler id. registerDefaultDiscovery is now nullable, and doDiscoveryLocal still does discoveryUpstreamDTO.setDiscoveryHandlerId(discoveryHandlerId) without a check (AbstractShenyuClientRegisterServiceImpl:316). Today it is safe only because the single caller returns early on null. Since the method is protected and now sits behind a nullable API, add a guard so a future second caller cannot insert upstreams with a null handler id.

  2. The URI path resolves the default handler twice. registerURIWithLock calls registerDefaultDiscovery to decide whether a remote discovery owns the selector, then doDiscoveryLocal calls it again and pays two FOR UPDATE reads plus the handler/discovery lookups inside the same transaction. Passing the resolved handler id into doDiscoveryLocal (or splitting "resolve" from "ensure") removes a redundant round-trip on the registration hot path.

  3. Fallback detection rests on a name convention. isDefaultLocalDiscovery matches discoveryLevel == SELECTOR && type == local && discoveryName == "<plugin>_default_discovery". That is the name registerDefaultDiscovery has always written, so it works today, but any manually created selector-level local discovery with that name would be silently repointed and its row deleted. Either reserve/document the name, or key the decision on something structural (the fallback always has handler == "{}" and no configured listener node).

  4. A differing shared discovery is silently ignored. When a plugin-level discovery of the requested type already exists, the registering client's serverList is discarded in favour of the first one, with no log. Two clusters configured for the same plugin+nnamespace+type is a plausible misconfiguration, and it currently collapses silently — one WARN comparing serverList would save a debugging session.

  5. selector.handle is left stale after a takeover. The promote path clears discovery_upstream and the runtime cache but deliberately does not rewrite the legacy selector.handle. The integration test shows the gateway ends up with only the remote instance, so behaviour is correct today; it is worth stating in the PR/README that the stale handle is intentional, otherwise someone will "fix" it later.

  6. Nits: SelectorService#findByNameAndPluginNameAndNamespaceIdForUpdate now has no production caller (dead API — either drop it or keep it documented as external); the plugin row lock is also taken on the early-return "conflicting configuration" path where nothing is written; and pluginMapper.selectByNameForUpdate result is discarded, so a missing plugin silently skips the lock it was meant to provide.

LGTM otherwise. CI is still pending (11 checks), so I am not merging yet — I will re-check once the suite reports.

@Aias00

Aias00 commented Oct 21, 2026

Copy link
Copy Markdown
Contributor

/retest

@Aias00 Aias00 left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Re-approving: the only new commit since my last review is a merge with master, and the coordinated registration logic is unchanged.

The ordering still holds: the initial URI registration and the discovery configuration registration are reconciled before the selector is persisted, so a client cannot end up bound to a discovery handler that has not been created yet. The added selectByName/selectByIds mapper methods are namespace-scoped, and the tests cover both the immediate-activation and the reused-discovery paths.

@Aias00

Aias00 commented Oct 23, 2026

Copy link
Copy Markdown
Contributor

Approved, but the checks are red. All three failing jobs trace back to the same shared http e2e case:

  • build (shenyu-integrated-test-http) — CachePluginTest.testMemoryModle / testRedisModle expect <true> but get <false>
  • e2e-case (...shenyu-examples-http) — downstream of that case
  • e2e — the aggregate gate over the two above

None of them exercises the URI/discovery coordination this PR changes. Could you keep it rebased and re-run once the http e2e case is fixed upstream? The code itself still looks good to me.

@Aias00

Aias00 commented Oct 24, 2026

Copy link
Copy Markdown
Contributor

@cloudbase-ai This PR is approved, but 9 checks are red: e2e, e2e-case (e2e-grpc-sync-compose, shenyu-e2e-case-grpc, :shenyu-examples-grpc), e2e-case (e2e-apache-dubbo-sync-compose, shenyu-e2e-case-apache-dubbo, :shenyu-examples-apache-du..., e2e-case (e2e-cluster-zookeeper-compose, shenyu-e2e-case-cluster, :shenyu-examples-http) (+5 more).

I inspected these lanes today and they fail for infrastructure reasons rather than on the diff — the integrated-test jobs die while starting their containers (Elasticsearch /_cat/health returning 000 until the job is cancelled), and the http lane also hits the known CachePluginTest.testMemoryModle / testRedisModle flake. The same jobs are red on master and on unrelated PRs right now.

Please rebase onto master and push (or ask for a re-run) so the checks execute again. If a case is still failing on a fresh run, paste the test name and I'll look properly.

@Aias00

Aias00 commented Oct 26, 2026

Copy link
Copy Markdown
Contributor

Hi @cloudbase-ai, this PR is already approved, but it currently cannot be merged. Here is the blocker and what is needed:

Blocker
This PR is approved but blocked by two distinct things:

  1. build (shenyu-integrated-test-http) fails in CachePluginTest.testRedisModle / testMemoryModle (AssertionFailedError: expected: <true> but was: <false> at CachePluginTest.java:73). The same failure reproduces on unrelated PRs (fix(instance): align persisted state labels across admin and dashboard #7889, fix(admin): preserve HTTPS for upstream Swagger documents #7895), so it is a systemic/flaky cache-integration test, not caused by your diff.
  2. The e2e-http case fails with a Maven DependencyResolutionException on a stale base.

What we need
Please rebase onto current master and re-run; the CachePluginTest failure should be treated as infra flakiness unless it reproduces in isolation.

Once the branch is up to date and CI is green, we will squash-merge it. Thanks!

@Aias00

Aias00 commented Oct 26, 2026

Copy link
Copy Markdown
Contributor

Root cause found for the build (shenyu-integrated-test-http) failure, and it is not in your branch — no code change needed on your side.

The job fails on:

CachePluginTest.testMemoryModle:54 expected: <true> but was: <false>
CachePluginTest.testRedisModle:73  expected: <true> but was: <false>

I traced this across three unrelated PRs (#7917, #7895, #7888) and all three fail on exactly the same two assertions, so it is not specific to your change. CachePluginTest.testPass() fetches /http/test/cache twice with a 2s gap and asserts the two response bodies match; the rule handle sets timeoutSeconds=3000, so a cache hit is expected. The logs show every response carrying a fresh timestamp ("msg":"Fri Oct 23 13:07:54 UTC 2026", then 13:07:55, then 13:07:56, then 13:07:58) — the cache never serves the stored entry. The test itself has not changed since 2022, so this is a cache-plugin regression introduced by the 10-23 / 10-24 cache work (#7969 coalesce concurrent misses, then #8079 Vary replay).

It is already fixed on master. The same job passes on newer heads:

Your branch is behind master (this PR: 58 commits), which is why it still carries the broken cache code.

What to do: rebase onto current master and let CI re-run. That should clear this job without touching your diff. If it still fails after the rebase, the regression is genuinely yours and I will dig in again.


Additional note for this PR: the e2e and e2e-case (...) failures are a separate infrastructure issue, not the cache bug. The log ends with:

[ERROR] dependency: org.apache.shenyu:shenyu-spring-boot-starter-client-springmvc:jar:2.7.2-SNAPSHOT (compile)
[ERROR] 	Could not find artifact org.apache.shenyu:shenyu-spring-boot-starter-client-springmvc:jar:2.7.2-SNAPSHOT

The e2e jobs consume the SNAPSHOT artifacts produced by build_local_repo, and that artifact was not available to this run. Same rebase-and-retry applies.

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

[Question] Conflict Between URIRegister and DiscoveryConfigRegister During Initial Service Registration

2 participants