Repository navigation
fix(admin): coordinate initial URI and discovery configuration registration - #7888
cloudbase-ai wants to merge 2 commits into
Conversation
…ration Signed-off-by: cloudbase-ai <[email protected]>
Aias00
left a comment
There was a problem hiding this comment.
Review: coordinate initial URI and configured discovery registration
This is a solid fix, and I checked the mechanism against the schema and the surrounding code rather than the description alone.
What I verified
- The takeover logic is correct. For a selector whose handler still points at the automatic LOCAL fallback,
bindingDiscoveryrepoints the existing handler (updateSelective) to the configured discovery, keeps thediscovery_relrow untouched, deletes thediscovery_upstreamrows by handler id, and only then removes the unreferenced fallbackdiscoveryrow. Relation/handler identity is genuinely preserved, so nothing on the gateway side sees a new selector binding. - The empty-registry case is covered. Clearing runtime state cannot rely on
fetchAllalone:fetchAllno-ops whendiscoveryServiceCachehas no entry for the discovery id, soremoveSelectorUpstreamis what guarantees the DELETE event fires. Calling it beforecreateDiscoveryis the right order — the DELETE lands, then the remote snapshot overwrites it. - Lock order is consistent.
registerDiscoveryConfiglocks selector → plugin;registerDefaultDiscoveryandregisterURIWithLocklock only the selector. Nothing in the new code takes the plugin lock before a selector lock, andSelectorService#findByNameAndPluginNameAndNamespaceIdForUpdate(plugin lock first) is no longer on either path. - The lock footprint is actually narrow, which is the part I expected to be a problem.
selectorhasidx_selector_ns_plugin_name (namespace_id, plugin_id, selector_name), sofindByNameAndPluginNameAndNamespaceIdForUpdatenext-key-locks a single record instead of scanning.selectBySelectorIdForUpdateresolves throughidx_discovery_rel_selectorand joinsdiscovery_handlerby primary key, so it locks one rel row plus one handler row.selectByPluginNameAndLevelAndNamespaceIdAndTypeForUpdateusesidx_discovery_ns_plugin. This matters becauseFOR UPDATEon an unindexed predicate would have serialized every registration in the cluster — it does not. registerDefaultDiscoveryreturningnullis handled at the only production call site.registerURIWithLockreturns early beforebuildHandle/updateSelective, so no local upstream or handle mutation happens once a remote discovery owns the selector. TheShenyuExceptionit now throws on namespace/plugin mismatch cannot fire on the URI path, because the selector is resolved by plugin name in the same statement.- Transaction and event ordering hold:
nativeCreateOrUpdateis a plain DB write with no event publication,updateSelectivegoes through the dispatcher'safterCommit, anddoDiscoveryLocalnow registers its publish after commit, so nothing is delivered for a rolled-back registration.testRolledBackUriRegistrationLeavesNoDiscoveryBindingpins exactly that.
The tests are unusually good for this area: arrival-order parameterization, two concurrent writers, an empty registry, rollback, and the independence assertion with a held selector lock all target real failure modes rather than coverage.
Non-blocking comments
-
doDiscoveryLocalno longer tolerates anullhandler id.registerDefaultDiscoveryis now nullable, anddoDiscoveryLocalstill doesdiscoveryUpstreamDTO.setDiscoveryHandlerId(discoveryHandlerId)without a check (AbstractShenyuClientRegisterServiceImpl:316). Today it is safe only because the single caller returns early onnull. Since the method isprotectedand now sits behind a nullable API, add a guard so a future second caller cannot insert upstreams with a null handler id. -
The URI path resolves the default handler twice.
registerURIWithLockcallsregisterDefaultDiscoveryto decide whether a remote discovery owns the selector, thendoDiscoveryLocalcalls it again and pays twoFOR UPDATEreads plus the handler/discovery lookups inside the same transaction. Passing the resolved handler id intodoDiscoveryLocal(or splitting "resolve" from "ensure") removes a redundant round-trip on the registration hot path. -
Fallback detection rests on a name convention.
isDefaultLocalDiscoverymatchesdiscoveryLevel == SELECTOR && type == local && discoveryName == "<plugin>_default_discovery". That is the nameregisterDefaultDiscoveryhas always written, so it works today, but any manually created selector-level local discovery with that name would be silently repointed and its row deleted. Either reserve/document the name, or key the decision on something structural (the fallback always hashandler == "{}"and no configured listener node). -
A differing shared discovery is silently ignored. When a plugin-level discovery of the requested type already exists, the registering client's
serverListis discarded in favour of the first one, with no log. Two clusters configured for the same plugin+nnamespace+type is a plausible misconfiguration, and it currently collapses silently — one WARN comparingserverListwould save a debugging session. -
selector.handleis left stale after a takeover. The promote path clearsdiscovery_upstreamand the runtime cache but deliberately does not rewrite the legacyselector.handle. The integration test shows the gateway ends up with only the remote instance, so behaviour is correct today; it is worth stating in the PR/README that the stale handle is intentional, otherwise someone will "fix" it later. -
Nits:
SelectorService#findByNameAndPluginNameAndNamespaceIdForUpdatenow has no production caller (dead API — either drop it or keep it documented as external); the plugin row lock is also taken on the early-return "conflicting configuration" path where nothing is written; andpluginMapper.selectByNameForUpdateresult is discarded, so a missing plugin silently skips the lock it was meant to provide.
LGTM otherwise. CI is still pending (11 checks), so I am not merging yet — I will re-check once the suite reports.
|
/retest |
Signed-off-by: cloudbase-ai <[email protected]>
Aias00
left a comment
There was a problem hiding this comment.
Re-approving: the only new commit since my last review is a merge with master, and the coordinated registration logic is unchanged.
The ordering still holds: the initial URI registration and the discovery configuration registration are reconciled before the selector is persisted, so a client cannot end up bound to a discovery handler that has not been created yet. The added selectByName/selectByIds mapper methods are namespace-scoped, and the tests cover both the immediate-activation and the reused-discovery paths.
|
Approved, but the checks are red. All three failing jobs trace back to the same shared http e2e case:
None of them exercises the URI/discovery coordination this PR changes. Could you keep it rebased and re-run once the http e2e case is fixed upstream? The code itself still looks good to me. |
|
@cloudbase-ai This PR is approved, but 9 checks are red: I inspected these lanes today and they fail for infrastructure reasons rather than on the diff — the Please rebase onto |
|
Hi @cloudbase-ai, this PR is already approved, but it currently cannot be merged. Here is the blocker and what is needed: Blocker
What we need Once the branch is up to date and CI is green, we will squash-merge it. Thanks! |
|
Root cause found for the The job fails on: I traced this across three unrelated PRs (#7917, #7895, #7888) and all three fail on exactly the same two assertions, so it is not specific to your change. It is already fixed on master. The same job passes on newer heads:
Your branch is behind master (this PR: 58 commits), which is why it still carries the broken cache code. What to do: rebase onto current master and let CI re-run. That should clear this job without touching your diff. If it still fails after the rebase, the regression is genuinely yours and I will dig in again. Additional note for this PR: the The e2e jobs consume the SNAPSHOT artifacts produced by |
URI registration can create an automatic LOCAL discovery before the client's Zookeeper/Nacos configuration arrives. Admin then reuses the LOCAL handler without rebinding it, so the configured discovery never owns the selector. Concurrent URI writers can also create duplicate handler/relation rows.
Fixes #5839.
Validation:
verify: 2,865 tests, 0 failures, 0 errors, 6 skipped; Checkstyle and RAT passed.The registration tests disable independent periodic HTTP health checking to isolate registration ordering; #7881 tracks that separate health-cache path. No two-Admin deployment or full Bootstrap request E2E is claimed. Existing duplicate relations are not automatically merged. No schema migration or dependency change is required; Oracle/OceanBase/openGauss registration integration was not executed locally.