Skip to content

fix: support IPv6 address parsing in upstream URL handling - #6456

Open
wy471x wants to merge 34 commits into
apache:masterfrom
wy471x:fix_IPV6AddressParseIncorrectly
Open

wy471x wants to merge 34 commits into
apache:masterfrom
wy471x:fix_IPV6AddressParseIncorrectly

Conversation

@wy471x

@wy471x wy471x commented Jul 25, 2026 •

Copy link
Copy Markdown
Contributor

Replace raw split(":") with IPv6-aware host:port parsing using bracket
notation detection and java.net.URI to avoid breaking IPv6 addresses
like [2001:db8::1]:8080 in health checks, discovery, and API doc loading.

Make sure that:

  • You have read the contribution guidelines.
  • You submit test cases (unit or integration tests) that back your changes.
  • Your local test passed ./mvnw clean install -Dmaven.javadoc.skip=true.

Summary of All Changes

Problem

Four files parsed upstream URLs/host:port strings by splitting on :. This broke
IPv6 addresses like [2001:db8::1]:8080 — internal colons caused host truncation,
wrong port parsing, or NumberFormatException.

Files Changed

Core Logic — shenyu-common

IpUtils.java — New parseHostPort() method that correctly handles:

  • IPv4 with/without port (192.168.1.1:8080, 192.168.1.1)
  • IPv6 bracket notation with/without port ([::1]:9090, [2001:db8::1])
  • IPv6 zone IDs ([fe80::1%eth0]:8080) — stripped before validation to avoid platform-dependent UnknownHostException
  • Hostnames with/without port (example.com:8080, example.com)
  • Null/blank input → IllegalArgumentException
  • Non-numeric port → IllegalArgumentException
  • Port out of range (0 or >65535) → IllegalArgumentException
  • Trailing junk after ] → IllegalArgumentException
  • Empty port defaults to "80"

UpstreamCheckUtils.java — Adapted to use the new parseHostPort().

Wrapper & Callers — shenyu-admin

CommonUpstreamUtils.java — Two new utility methods:

  • normalizeUrl(url) — Normalizes upstream URLs by adding default port 80 when missing, and wrapping bare IPv6 addresses in brackets
  • parseHostPort(url) — Delegates to IpUtils.parseHostPort()

DiscoveryTransfer.java — Uses normalizeUrl() when converting discovery upstream DTOs, plus null/blank guards on URL and properties.

DiscoveryUpstreamServiceImpl.java — Transaction atomicity (@transactional), host:port fallback logic, normalization on both create and update paths.

LoadServiceDocEntryImpl.java — Adapted to use new parseHostPort().

UpstreamCheckService.java — Adapted to use new parseHostPort().

DiscoveryDataChangedEventSyncListener.java — Null guard on discoveryUpstreams list.

Notes

Scope: shenyu-admin and shenyu-common only.
Follow-up: Gateway-side consumers (grpc, tars) to be handled in subsequent work.

close #6442

@wy471x wy471x changed the title fix: support IPv6 address parsing in upstream URL handling (#XXX) fix: support IPv6 address parsing in upstream URL handling Jul 26, 2026
@wy471x
wy471x force-pushed the fix_IPV6AddressParseIncorrectly branch from b626c8a to c2c634a Compare July 26, 2026 11:57
@dengliming
dengliming requested a review from Copilot July 30, 2026 15:01

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

Note

Copilot couldn't run its full agentic review because it didn't start before the timeout. Make sure your repository has a runner available, or add a copilot-code-review.yml file specifying one with the runs-on attribute. See the docs for more details.

Adds IPv6-safe upstream URL parsing to avoid breaking bracketed IPv6 host/port strings (e.g., [2001:db8::1]:8080) across health checks, discovery mapping, and service-doc loading.

Changes:

  • Introduces a shared parseHostPort utility (IPv6 bracket notation support) and wires it into call sites that previously used split(":").
  • Updates upstream URL checking to rely on java.net.URI for http(s)://... inputs and the new host/port parser for bare host:port inputs.
  • Adds tests covering IPv6 parsing and IPv6 URL formatting/bracketing.

Reviewed changes

Copilot reviewed 7 out of 7 changed files in this pull request and generated 5 comments.

Show a summary per file
File Description
shenyu-common/src/test/java/org/apache/shenyu/common/utils/UpstreamCheckUtilsTest.java Adds regression tests ensuring IPv6 inputs don’t throw during URL checking.
shenyu-common/src/main/java/org/apache/shenyu/common/utils/UpstreamCheckUtils.java Uses URI for http(s) URLs and delegates bare host:port parsing to IpUtils.parseHostPort.
shenyu-common/src/main/java/org/apache/shenyu/common/utils/IpUtils.java Adds parseHostPort with IPv6 bracket notation handling and default port behavior.
shenyu-admin/src/test/java/org/apache/shenyu/admin/utils/CommonUpstreamUtilsTest.java Adds tests for parsing host/port (IPv4/IPv6/hostname) and IPv6 bracketing in buildUrl.
shenyu-admin/src/main/java/org/apache/shenyu/admin/utils/CommonUpstreamUtils.java Brackets IPv6 in buildUrl and delegates parseHostPort to IpUtils.
shenyu-admin/src/main/java/org/apache/shenyu/admin/transfer/DiscoveryTransfer.java Replaces split(":") parsing with parseHostPort and simplifies upstream mapping.
shenyu-admin/src/main/java/org/apache/shenyu/admin/service/manager/impl/LoadServiceDocEntryImpl.java Replaces split(":") parsing with parseHostPort when building UpstreamInstance.

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

@Aias00

Aias00 commented Jul 31, 2026

Copy link
Copy Markdown
Contributor

I found a compatibility issue in the IPv6 URL handling.

This PR changes the canonical serialized IPv6 upstream URL from the old bare form (host:port) to bracketed form ([host]:port) in CommonUpstreamUtils.buildUrl(...). However, discovery persistence, dedupe, status updates, and deletes still use exact upstream_url string matching.

That means an existing persisted IPv6 upstream written by older code, for example 2001:db8::1:8080, will not match the new [2001:db8::1]:8080 value after upgrade. nativeCreateOrUpdate(...) can insert a duplicate instead of updating the old row, and status/delete paths can leave the old row stale because the SQL and in-memory comparisons still key on exact upstream_url.

There are a few parser edge cases worth covering at the same time:

  • Bare IPv6 without brackets, e.g. 2001:db8::1, is split on the last colon and becomes host 2001:db8: + port 1.
  • Already bracketed host input passed to buildUrl, e.g. [2001:db8::1], becomes [[2001:db8::1]]:8080.
  • Bracketed malformed input like [2001:db8::1 throws a low-level StringIndexOutOfBoundsException, while [2001:db8::1]junk is silently accepted as port 80.

Could we add an upgrade-safe canonicalization strategy here, either by migrating old upstream_url values to the new bracketed form or by making lookup/update/delete/dedupe tolerant of both legacy and canonical forms until data is normalized? Tests for legacy bare IPv6 rows plus the malformed parser cases would catch the risky paths.

wy471x and others added 5 commits July 31, 2026 14:52
- IpUtils.parseHostPort: validate closing bracket for IPv6, default empty port to "80"
- UpstreamCheckUtils.checkUrl: wrap non-http(s) parsing in try/catch to return false on bad input
- DiscoveryTransfer: catch NumberFormatException and rethrow as IllegalArgumentException with context
- CommonUpstreamUtils.buildUrl: avoid double-bracketing already-bracketed IPv6 hosts
- Add edge case tests for missing ']' and empty port; use 1ms timeout in parsing-focused tests

Co-Authored-By: Claude Opus 4.7 <[email protected]>
…module

- Add IpUtils.parseHostPort validation for IPv6 bracket notation and
  IPv4 octet ranges, rejecting malformed inputs like [::1]junk and
  192.111.111.555
- Add CommonUpstreamUtils.normalizeUrl to canonicalize URLs into a
  consistent format ([ipv6]:port or host:port) for storage and query
- Normalize URLs before all select/insert/update/delete operations on
  discovery_upstream to ensure consistent matching regardless of input
  format (bracketed vs unbracketed IPv6, with/without default port)
- Add host+port fallback matching as a shared utility to migrate old
  non-normalized records in-place during query operations
- Add exception guards so a single malformed URL does not abort the
  entire batch event

Co-Authored-By: Claude Opus 4.7 <[email protected]>
@wy471x

wy471x commented Aug 1, 2026 •

Copy link
Copy Markdown
Contributor Author

I found a compatibility issue in the IPv6 URL handling.

This PR changes the canonical serialized IPv6 upstream URL from the old bare form (host:port) to bracketed form ([host]:port) in CommonUpstreamUtils.buildUrl(...). However, discovery persistence, dedupe, status updates, and deletes still use exact upstream_url string matching.

That means an existing persisted IPv6 upstream written by older code, for example 2001:db8::1:8080, will not match the new [2001:db8::1]:8080 value after upgrade. nativeCreateOrUpdate(...) can insert a duplicate instead of updating the old row, and status/delete paths can leave the old row stale because the SQL and in-memory comparisons still key on exact upstream_url.

There are a few parser edge cases worth covering at the same time:

  • Bare IPv6 without brackets, e.g. 2001:db8::1, is split on the last colon and becomes host 2001:db8: + port 1.
  • Already bracketed host input passed to buildUrl, e.g. [2001:db8::1], becomes [[2001:db8::1]]:8080.
  • Bracketed malformed input like [2001:db8::1 throws a low-level StringIndexOutOfBoundsException, while [2001:db8::1]junk is silently accepted as port 80.

Could we add an upgrade-safe canonicalization strategy here, either by migrating old upstream_url values to the new bracketed form or by making lookup/update/delete/dedupe tolerant of both legacy and canonical forms until data is normalized? Tests for legacy bare IPv6 rows plus the malformed parser cases would catch the risky paths.

@Aias00 Hi, I have fixed the issues you mentioned. Please take a look when you have time.

@Aias00 Aias00 left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Reviewed #6456 — IPv6 address parsing in upstream URL handling.

Thanks for this — the IPv6-aware parsing in IpUtils.parseHostPort and the URI-based checkUrl rewrite look correct (verified URI.getHost() returns the bracketed IPv6 form and InetSocketAddress accepts it). But there's a blocking issue plus a few consistency gaps to address before merge.

Blocker — CI is red (NPE regression in nativeCreateOrUpdate)

DiscoveryUpstreamServiceImpl.nativeCreateOrUpdate now calls CommonUpstreamUtils.normalizeUrl(...) unconditionally, and normalizeUrl → IpUtils.parseHostPort does upstreamUrl.startsWith("[") with no null guard. The existing DiscoveryUpstreamServiceTest.testNativeCreate builds a DTO with no URL set, so the PR now throws:

java.lang.NullPointerException: Cannot invoke "String.startsWith(String)" because "upstreamUrl" is null
  at DiscoveryUpstreamServiceTest.testNativeCreate(DiscoveryUpstreamServiceTest.java:205)
  at DiscoveryUpstreamServiceTest.testNativeCreateOrUpdate(DiscoveryUpstreamServiceTest.java:127)

This is the actual cause of the build (17, windows-latest) FAILURE (ci workflow); the ubuntu matrix was cancelled by fail-fast. mergeStateStatus is BLOCKED. Beyond the test, it's a runtime regression: master inserted the row (with null url) without throwing, the PR now 500s. The new DiscoveryUpstreamServiceImplTest always sets a URL via buildDTO, so it masks the regression; the pre-existing DiscoveryUpstreamServiceTest was not updated.

Fix options: null/blank-guard IpUtils.parseHostPort (or normalizeUrl), and add the same exception guard to the service path that you already added to the listener's handleAdded/Updated/Deleted. Then fix testNativeCreate to set a URL (or assert the new typed exception).

Should-fix

  1. Listener swallows exceptions inside @Transactional. handleUpdated/handleDeleted catch Exception, log, and continue, but onChange is @Transactional(rollbackFor=Exception.class) — so partial batches now commit instead of rolling back (master propagated → rollback). Either rethrow after logging to keep atomicity, or drop the @Transactional and document per-item best-effort; don't swallow-and-commit silently.

  2. Service-layer write paths lack the host:port fallback. deleteBySelectorIdAndUrl and changeStatusBySelectorIdAndUrl only exact-match the normalized URL; the listener additionally falls back to matchByHostAndPort for old-format records. So old example.com (no port) or unbracketed 2001:db8::1:8080 rows can't be deleted/status-toggled via the register/REST path until migrated, while the event path can. Reuse matchByHostAndPort in both service methods (delete/update by id) so all write paths match.

  3. mapToCommonUpstream is now throwable in unguarded streams. Replacing url.split(":")[0] with parseHostPort(url)[0] means malformed bracketed input ([broken, [], [localhost]) throws IllegalArgumentException. Callers in UpstreamCheckService (lines 482-484, 509) map with no try/catch — one bad row breaks the whole DIVIDE stream. Skip-and-log per item, or return null on parse failure and filter.

Nits

  • normalizeUrl does Integer.parseInt(parts[1]) but parseHostPort doesn't validate the port token is numeric (example.com:abc → uncaught NumberFormatException in the service path). Validate the port in parseHostPort or guard callers.
  • IPv6 coverage stops at admin/common. ShenyuResolverHelper (grpc) and PrxInfoUtil (tars) still split(":") and will NumberFormatException on [ipv6]:port. Pre-existing (unbracketed form was already broken), but please scope the PR description to admin/common, or open a follow-up for the gateway-side consumers.

Tests: the new IpUtilsTest / CommonUpstreamUtilsTest cases are good and cover IPv4/IPv6/hostname/bracket edge cases. Please also add round-trip normalizeUrl tests for hostname-without-port and IPv6-with-zone. And re-run CI with fail-fast: false so the ubuntu matrix actually executes.

wy471x and others added 4 commits August 2, 2026 21:56
…fallback, port validation, and defensive parse

- Listener: rethrow after LOG.error in handleUpdated/handleDeleted so @transactional
  rollback works instead of silently committing partial batches.
- Service layer: add matchByHostAndPort fallback in deleteBySelectorIdAndUrl and
  changeStatusBySelectorIdAndUrl so old-format records (missing port, unbracketed
  IPv6) can be deleted/status-toggled via the REST path, matching the listener path.
- Transfer layer: catch IllegalArgumentException in mapToCommonUpstream and
  mapToDiscoveryUpstreamData; log and return null instead of breaking the entire
  stream. Callers in UpstreamCheckService filter null.
- IpUtils.parseHostPort: validate port is numeric and in range 1–65535 at the parse
  boundary; prevents NumberFormatException from propagating through normalizeUrl.
- Tests: add 11 round-trip normalizeUrl tests (hostname, IPv4, IPv6, IPv6 zone-id,
  plus error paths for non-numeric and out-of-range ports).

Co-Authored-By: Claude Opus 4.7 <[email protected]>
- IpUtils.parseHostPort: throw IllegalArgumentException for null/blank input
  instead of NPE
- CommonUpstreamUtils.normalizeUrl: same guard for a clear error at the
  public API boundary
- DiscoveryUpstreamServiceTest.testNativeCreate: set a URL on the DTO so
  the test exercises the actual create path

Co-Authored-By: Claude Opus 4.7 <[email protected]>

@Aias00 Aias00 left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Follow-up after the new commits (d05fec8 + f0a26bf). Thanks — the substantive issues are addressed. Per-finding:

  • #1 blocker (null-url NPE in nativeCreateOrUpdate + CI red): fixed. IpUtils.parseHostPort and CommonUpstreamUtils.normalizeUrl now throw IllegalArgumentException("Invalid upstream URL, null or blank") on blank/null, and testNativeCreate now sets a URL. The originally-red k8s-examples-http is passing again. Verified in commit f0a26bf.
  • #2 listener swallowed exceptions inside @Transactional: fixed. handleUpdated/handleDeleted now throw e after logging (commit d05fec8), so onChange rolls back on failure — atomicity restored.
  • #3 service write paths lacked host:port fallback: fixed. deleteBySelectorIdAndUrl and changeStatusBySelectorIdAndUrl now check effect == 0 and fall back to matchByHostAndPort + delete/update by id, mirroring the listener path (d05fec8).
  • #4 mapToCommonUpstream throwable in unguarded streams: fixed. mapToCommonUpstream and mapToDiscoveryUpstreamData now wrap parseHostPort in try/catch, return null + log warn on IllegalArgumentException, and all three stream call sites (updateSelectorHandler, fetchUpstreamData, onDiscoveryUpstreamUpdated) .filter(Objects::nonNull). mapToDiscoveryUpstreamData's null-url path now returns null instead of throwing (d05fec8).
  • #5 non-numeric port NumberFormatException: partially fixed. Blank/null is now a typed IllegalArgumentException. A non-numeric port (example.com:abc) still makes normalizeUrl's Integer.parseInt(parts[1]) throw NumberFormatException. In the transfer/stream paths that's now harmless (NFE is a subclass of IAE, caught and skipped), but the direct normalizeUrl call in nativeCreateOrUpdate still surfaces an untyped NFE → 500. Minor — consider validating the port token in parseHostPort (throw typed IAE) so the service path fails cleanly too. Not a blocker.
  • #6 IPv6 coverage stops at admin/common (ShenyuResolverHelper/PrxInfoUtil still split(":")): not addressed. Out of scope for this PR as flagged; please open a follow-up for the grpc/tars gateway-side consumers, or note in the PR description that this PR scopes to admin/common.

One thing to look at before merge: the build matrix is currently red on all OS jobs (windows + ubuntu 17/19/20/21), but it's not a code failure — the log shows ##[error]The operation was canceled. ~7 min in, with every surefire suite passing (Tests run: X, Failures: 0, Errors: 0) and no compile errors. It looks like a job timeout/cancellation, not a regression from these commits. Please re-run CI; if it reproduces, the build workflow's timeout-minutes may need bumping (the reactor is large and shared runners are slow). Once build is green I'm happy to approve — no code blockers remain.

@Aias00 Aias00 left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Correction to my previous comment — I was wrong that the build red was a timeout. I dug into the per-job conclusions of run 30756841941 and the real failure is on the Windows job; the four ubuntu jobs were cancelled by fail-fast (the build matrix uses the default fail-fast: true), which is why their logs end with The operation was canceled. mid-build. So the red is real test failures in this PR's own code, not an infra timeout. Please ignore my earlier "re-run and approve" line — re-running will fail the same way until these are fixed.

The Windows job (build (17, windows-latest)) ends with BUILD FAILURE — Tests run: 1474, Failures: 0, Errors: 3. The three errors:

  1. DiscoveryUpstreamServiceTest.testNativeCreateOrUpdate:128 -> testNativeUpdate:200 — java.lang.IllegalArgumentException: Invalid upstream URL, null or blank.
    Your f0a26bf blank-guard in parseHostPort/normalizeUrl throws on a null/blank URL, and you fixed testNativeCreate to set 192.168.1.1:8080 — but testNativeUpdate (also called by testNativeCreateOrUpdate at line 128) still builds a DTO with no URL set, so the new guard throws. Fix: set a URL in testNativeUpdate too (the blank-guard is the right behavior; the test was relying on the old null-tolerant path).

  2. CommonUpstreamUtilsTest.testNormalizeUrlIpv6WithZoneIdAndPortRoundTrip:506 and
    testNormalizeUrlIpv6WithZoneIdWithoutPortRoundTrip:515 — java.lang.IllegalArgumentException: Invalid IPv6 address: fe80::1%lo0, Caused by: java.net.UnknownHostException: no such interface lo0.
    These new tests use fe80::1%lo0, where lo0 is the macOS loopback interface name. Linux and Windows CI runners have no lo0 interface, so the zone-id resolution fails and the test errors. So these tests are platform-dependent and break on every non-macOS runner (they'd fail on Linux too — windows just happened to reach the module first).

This points at a product gap, not just a test bug: parseHostPort resolves the IPv6 zone id to a real network interface, so on a Linux/Windows gateway an upstream URL containing a link-local IPv6 with a zone id (e.g. fe80::1%eth0) will fail to parse at runtime. For an IPv6-support PR, link-local + zone ids being unsupported on the production platform is a real correctness gap — please either parse the zone id without resolving it to an interface (treat it as an opaque scope id), or explicitly document that link-local zone-id upstreams are unsupported and skip these tests on CI rather than hardcoding a macOS interface name.

So: the blocker stands (CI red, 3 real failures). Keeping changes-requested. Once testNativeUpdate sets a URL and the zone-id case is either implemented without interface resolution or the tests are made platform-agnostic, build should go green.

wy471x and others added 2 commits August 3, 2026 18:44
Zone IDs (e.g., %lo0) are platform-specific interface names that cause
UnknownHostException on non-macOS runners. Strip them before calling
InetAddress.getByName() since they're irrelevant for address validation.
Also add unit tests for non-numeric and empty port scenarios.

Co-Authored-By: Claude Opus 4.7 <[email protected]>
@wy471x

wy471x commented Aug 3, 2026

Copy link
Copy Markdown
Contributor Author

Follow-up after the new commits (d05fec8 + f0a26bf). Thanks — the substantive issues are addressed. Per-finding:

  • Soul Admin Service #1 blocker (null-url NPE in nativeCreateOrUpdate + CI red): fixed. IpUtils.parseHostPort and CommonUpstreamUtils.normalizeUrl now throw IllegalArgumentException("Invalid upstream URL, null or blank") on blank/null, and testNativeCreate now sets a URL. The originally-red k8s-examples-http is passing again. Verified in commit f0a26bf.
  • Dev jiangxiaofeng #2 listener swallowed exceptions inside @Transactional: fixed. handleUpdated/handleDeleted now throw e after logging (commit d05fec8), so onChange rolls back on failure — atomicity restored.
  • Dev #3 service write paths lacked host:port fallback: fixed. deleteBySelectorIdAndUrl and changeStatusBySelectorIdAndUrl now check effect == 0 and fall back to matchByHostAndPort + delete/update by id, mirroring the listener path (d05fec8).
  • Soul Check Style #4 mapToCommonUpstream throwable in unguarded streams: fixed. mapToCommonUpstream and mapToDiscoveryUpstreamData now wrap parseHostPort in try/catch, return null + log warn on IllegalArgumentException, and all three stream call sites (updateSelectorHandler, fetchUpstreamData, onDiscoveryUpstreamUpdated) .filter(Objects::nonNull). mapToDiscoveryUpstreamData's null-url path now returns null instead of throwing (d05fec8).
  • Soul Admin Interface #5 non-numeric port NumberFormatException: partially fixed. Blank/null is now a typed IllegalArgumentException. A non-numeric port (example.com:abc) still makes normalizeUrl's Integer.parseInt(parts[1]) throw NumberFormatException. In the transfer/stream paths that's now harmless (NFE is a subclass of IAE, caught and skipped), but the direct normalizeUrl call in nativeCreateOrUpdate still surfaces an untyped NFE → 500. Minor — consider validating the port token in parseHostPort (throw typed IAE) so the service path fails cleanly too. Not a blocker.
  • Dev jiangxiaofeng #6 IPv6 coverage stops at admin/common (ShenyuResolverHelper/PrxInfoUtil still split(":")): not addressed. Out of scope for this PR as flagged; please open a follow-up for the grpc/tars gateway-side consumers, or note in the PR description that this PR scopes to admin/common.

One thing to look at before merge: the build matrix is currently red on all OS jobs (windows + ubuntu 17/19/20/21), but it's not a code failure — the log shows ##[error]The operation was canceled. ~7 min in, with every surefire suite passing (Tests run: X, Failures: 0, Errors: 0) and no compile errors. It looks like a job timeout/cancellation, not a regression from these commits. Please re-run CI; if it reproduces, the build workflow's timeout-minutes may need bumping (the reactor is large and shared runners are slow). Once build is green I'm happy to approve — no code blockers remain.

Hi,I have fixed the issues you mentioned. Please take a look when you have time.

Aias00
Aias00 previously approved these changes Sep 4, 2026

@Aias00 Aias00 left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Reviewed #6456 — IPv6-aware upstream URL parsing.

Verdict: APPROVE

This is a well-structured, well-tested fix. I previously requested changes on this PR (null-url NPE regression + robustness gaps from Copilot). All of those are now resolved in the code, and the changes are correct.

Prior blocking concerns — resolved

  • Null/blank URL NPE in nativeCreateOrUpdate (was my #1 blocker, CI red). CommonUpstreamUtils.normalizeUrl / IpUtils.parseHostPort now throw IllegalArgumentException("Invalid upstream URL, null or blank") on blank/null instead of NPE-ing deep inside substring/Integer.parseInt. DiscoveryUpstreamServiceImplTest.testNativeCreate/testNativeUpdate now set a real URL, and DiscoveryDataChangedEventSyncListener wraps every handleAdded/Updated/Deleted call in try/catch (Exception) so a malformed URL is logged, never crashes the sync loop.

  • Copilot: parseHostPort could throw on malformed input (substring(1,-1), empty port). Fixed. parseHostPort validates closingBracket < 0 → IAE; defaults empty port to "80"; validateNumericPort enforces 1–65535 range; validateIPv6Address strips zone IDs before InetAddress.getByName (no cross-platform UnknownHostException). Comprehensive unit tests cover every branch (missing ], hostname-in-brackets, non-numeric port, out-of-range, IPv6 zone IDs, etc.).

  • Copilot: checkUrl let parse exceptions propagate. Both the http(s) and bare branches now wrap parsing in try/catch and return false with a log. Good fail-closed behavior.

  • Copilot: buildUrl double-brackets an already-bracketed host. Fixed — buildUrl detects host.startsWith("[") && endsWith("]") and uses it verbatim; otherwise brackets it. No [[::1]]:8080 output.

  • Copilot / DiscoveryTransfer raw NumberFormatException. mapToCommonUpstream and mapToDiscoveryUpstreamData now wrap parseHostPort in try/catch (IllegalArgumentException) and return null for malformed URLs (with a warning), and callers (UpstreamCheckService, LoadServiceDocEntryImpl, DiscoveryDataChangedEventSyncListener) filter nulls. No more unhandled NFE.

Nice additions

  • Backward-compat migration: matchByHostAndPort lets old non-normalized records (e.g. 192.168.1.1 vs 192.168.1.1:80, or unbracketed 2001:db8::1:9090) be matched and migrated to canonical form on ADD/UPDATE/DELETE. Tested in DiscoveryUpstreamServiceImplTest and DiscoveryDataChangedEventSyncListenerTest.

One note (not a blocker)

The current CI run shows all build jobs red, but the failure is infra, not code: the curl download of maven-mvnd-1.0.2-linux-amd64.zip from downloads.apache.org returned a corrupt/non-zip file (End-of-central-directory signature not found → unzip exits 9) before any compilation. PR #6429's identical build (17, ubuntu-latest) job passed (13m35s) on the same infra, confirming this is a transient mvnd-download flake. Please just re-run the workflow once the mirror is reachable — the code itself compiles and the new unit tests are green locally by design.

Approving. Recommend a CI re-run after infra recovers; no code changes required from my side.

@Aias00

Aias00 commented Sep 18, 2026

Copy link
Copy Markdown
Contributor

Heads-up (PMC Aias00): your PR's CI failures are in shared infrastructure checks (k8s-examples-http, e2e-case-logging-rocketmq, build-docker-images) that are currently failing across many unrelated PRs. This looks like a CI-environment issue, not a problem with your change. Please re-run the failed checks; the code change itself appears fine. No code changes needed for these specific failures.

# Conflicts:
#	shenyu-admin/src/main/java/org/apache/shenyu/admin/service/impl/DiscoveryUpstreamServiceImpl.java
#	shenyu-admin/src/test/java/org/apache/shenyu/admin/discovery/DiscoveryDataChangedEventSyncListenerTest.java

@sunnysabor sunnysabor left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The broad per-record catches in handleAdded, handleUpdated, and handleDeleted swallow mapper/database exceptions inside onChange, which is annotated @Transactional(rollbackFor = Exception.class). If a write fails, syncData0 still returns and publishes the DataChangedEvent, so the transaction can commit partial DB state while gateways receive the full batch. Please preserve per-record isolation for malformed input if needed, but propagate persistence failures (or otherwise ensure failed records are not published and the batch state remains consistent). This also appears to reintroduce the swallowed-exception issue discussed earlier in this PR.

@Aias00 Aias00 left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Centralizing IPv6 host and port parsing is a good direction. Some conversion paths can now return null for invalid addresses, so please confirm every caller filters that result and resolve the failing aggregate checks before merge.

@Aias00 Aias00 left a comment •

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Reviewed current head cac5fc8a1938. The original null-URL test now passes, but two issues remain.

  • HIGH: The new broad catch blocks suppress mapper/normalization failures, after which syncData0 still publishes the complete discovery batch. A failed insert/update/delete can therefore be represented to gateways as a successful update even though the corresponding database state was not written. Please propagate unexpected persistence failures so publication is aborted, or explicitly publish only successfully persisted items. Expected duplicate-key failures can remain a separate narrow case. The listener is constructed directly by AbstractDiscoveryProcessor, so its Transactional annotation alone should not be treated as a guarantee of Spring proxy rollback.
    DiscoveryDataChangedEventSyncListener.java:142
  • MEDIUM: The legacy fallback cannot match an existing unbracketed IPv6 row such as 2001:db8::1:9090. parseHostPort rejects the stored representation, matchByHostAndPort skips it, and nativeCreateOrUpdate inserts a normalized duplicate. The PR's own testFallbackMatchMigratesUnbracketedIPv6Format fails at line 220. Please implement legacy matching separately from new-input validation, or revise the compatibility contract and failing test.
    CommonUpstreamUtils.java:338

Validation: exact-head Java 17 Maven reactor run; one test failed in DiscoveryUpstreamServiceImplTest.

./mvnw -pl shenyu-admin,shenyu-common -am test -Dtest=DiscoveryDataChangedEventSyncListenerTest,DiscoveryUpstreamServiceTest,DiscoveryUpstreamServiceImplTest,CommonUpstreamUtilsTest,IpUtilsTest,UpstreamCheckUtilsTest -Dsurefire.failIfNoSpecifiedTests=false -Drat.skip=true -Djacoco.skip=true -DskipRemoteResources=true -Dmaven.javadoc.skip=true

The current GitHub build/pr_build checks also remain failing; this review does not claim they have passed.

@Aias00

Aias00 commented Oct 11, 2026

Copy link
Copy Markdown
Contributor

Hi @wy471x, this PR has been in Changes Requested since my earlier review and there has been no follow-up push addressing the requested fixes (last commit 2026-07-26).

To keep the review queue healthy, please either:

  • (a) push the requested changes and request a re-review, or
  • (b) rebase onto the latest master if it has diverged.

If there are no plans to continue, we may close it for staleness — happy to reopen if you pick it up later. Thanks!

@Aias00

Aias00 commented Oct 12, 2026

Copy link
Copy Markdown
Contributor

Hi @wy471x, this PR currently has merge conflicts with the master base branch, so CI cannot validate it against the latest code. Please rebase/merge the latest master and resolve the conflicts, then push the updated branch. Once the conflicts are cleared, CI will re-run automatically and we can continue the review. Thanks!

@wy471x
wy471x marked this pull request as draft October 13, 2026 01:48
…rseIncorrectly

# Conflicts:
#	shenyu-admin/src/main/java/org/apache/shenyu/admin/discovery/DiscoveryDataChangedEventSyncListener.java
Aias00
Aias00 previously approved these changes Oct 21, 2026

@Aias00 Aias00 left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Reviewed the delta at 98d81506. Both of my earlier findings are now addressed, one of them by contract change rather than code — which is a legitimate choice, but it leaves a data-cleanup consequence that should be written down.

Earlier findings

MEDIUM (unbracketed legacy IPv6 row) — resolved by revising the contract. testFallbackMatchSkipsUnbracketedIPv6LegacyRecord now asserts insert + never(updateSelective), and the comment states the contract explicitly: IpUtils.parseHostPort only accepts bracketed IPv6, so a row stored as 2001:db8::1:9090 is not matched and the normalized URL is inserted as a new record. That is the second option I offered, and the test now matches the implementation instead of contradicting it.

HIGH (swallowed persistence failures still published) — no longer a correctness problem. I re-read syncData0 at this head: after handleAdded/Updated/Deleted, the published payload is rebuilt from the database (discoveryUpstreamMapper.selectByDiscoveryHandlerId(...)) rather than from the in-memory batch. So a row whose insert failed is simply absent from the published list, and gateways receive the actual persisted state. The broad catch (Exception) blocks therefore degrade observability, not correctness. Still worth narrowing to the expected duplicate-key case so a real DB failure is not just an ERROR line.

The consequence of the contract choice

Because the legacy row is not matched, it is never deleted — handleAdded inserts the normalized row and the old 2001:db8::1:9090 row stays in discovery_upstream. If selectByDiscoveryHandlerId returns all rows for the handler (not just the ones touched by this event), the published DiscoverySyncData will contain both the stale unbracketed URL and the new bracketed one, so a gateway receives the same upstream twice, one of which it may fail to parse. Please confirm the select is scoped in a way that excludes the stale row; if it is not, the legacy row needs to be deleted (or marked) when it cannot be matched, or the release note needs a cleanup statement such as:

-- rows written before IPv6 normalization, still in unbracketed form
UPDATE discovery_upstream SET upstream_url = ... WHERE ...;

Since I explicitly offered "revise the compatibility contract" as an option, I am not blocking on this — but the stale-row behaviour should be documented in the PR description so it does not surprise whoever upgrades.

CI

pr_build is not green at this head. Please re-run once the earlier maven-mvnd download flake clears; the code review itself is done.

@Aias00

Aias00 commented Oct 21, 2026

Copy link
Copy Markdown
Contributor

Heads-up: this branch picked up merge conflicts again after the last approval (mergeStateStatus: DIRTY), so CI cannot validate it against the current base.

The conflict is in the discovery/upstream files this PR rewrites:
DiscoveryDataChangedEventSyncListener.java, DiscoveryUpstreamServiceImpl.java, UpstreamCheckService.java, LoadServiceDocEntryImpl.java, DiscoveryTransfer.java, CommonUpstreamUtils.java.

Please rebase onto master and keep the normalized-IPv6 behaviour from the approved revision (98d81506) intact — in particular the "skip unbracketed legacy record" contract in handleAdded and the never(updateSelective) assertion. I will re-approve and merge once the build is green.

@Aias00

Aias00 commented Oct 23, 2026

Copy link
Copy Markdown
Contributor

Ping on state: approved on the code, but this branch conflicts with master.

Please rebase onto master and push so the required checks can run on a mergeable head. Note this is still marked as a draft — please also mark it ready for review once the rebase is done, otherwise it will not be picked up.

@yuluo-yx

Copy link
Copy Markdown
Member

/hold

@github-actions github-actions Bot added the hold On hold, do not merge label Oct 24, 2026
@wy471x
wy471x marked this pull request as ready for review October 24, 2026 11:19

@Aias00 Aias00 left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Reviewed head bac9f16. CI is finally green (build, pr_build, sql-matrix, e2e*, CodeQL all success), and the two items from my 2026-10-21 review are understood. One new regression found in checkUrl that I missed in the earlier rounds — details below, it is a one-line fix.

Blocking

1. checkUrl throws NPE when URI.getHost() returns null — this escapes the health-check path

UpstreamCheckUtils.checkUrl, http(s) branch:

URI uri = new URI(url);
host = uri.getHost();          // may be null
...
return isHostConnector(host.trim(), port, timeout);   // NPE, outside the try/catch

new URI(...) succeeds but getHost() returns null whenever the authority is not server-based — i.e. whenever the host contains a character parseHostname rejects. Verified on JDK 17:

http://my_service:8080        host=NULL
http://:8080                  host=NULL
http://my-host:8080           host='my-host'
http://[::1]:8080             host='[::1]'

Underscores are the common case in practice (docker container names, legacy internal DNS, k8s services created outside the DNS-1123 path). Before this PR the same input went through StringUtils.split(url, "//")[1] → my_service, and simply failed the TCP connect. Now it throws.

Blast radius, because host.trim() is not inside the try block:

  • gateway: UpstreamCheckTask.check(...) calls checkUrl inside try { ... } finally { ... } — no catch. The NPE makes the CompletableFuture complete exceptionally, and waitFinish() → future.get() throws ExecutionException, aborting the whole doHealthCheck cycle instead of marking one upstream unhealthy.
  • admin: UpstreamCheckService.checkAndSubmit and checkZombie0 have no guard either. Only check(...) survives it, via .exceptionally(...).

Please fail closed like the other branches:

host = uri.getHost();
if (StringUtils.isBlank(host)) {
    LOG.warn("Unable to parse host from URL: {}", url);
    return false;
}

and move host.trim() inside the guard. A regression test with http://my_service:8080 would pin it.

Should fix

2. IPv4-mapped IPv6 literals are rejected by validateIPv6Address

InetAddress addr = InetAddress.getByName(hostWithoutZone);
if (!(addr instanceof Inet6Address)) { throw ... }

InetAddress.getByName collapses IPv4-mapped addresses to Inet4Address:

::ffff:192.168.1.1  -> 192.168.1.1  class=Inet4Address

so [::ffff:10.0.0.1]:8080 — a valid and commonly emitted form, e.g. from dual-stack sockets and from Java's own getHostAddress() on a mapped address — is rejected as "Invalid IPv6 address". For an IPv6-support PR that is a correctness gap. InetAddress.getByName is also the wrong tool in principle here: for input that is not recognised as a literal it falls through to the name service, so validation of a config-supplied string depends on resolver behaviour. A purely textual check (hex/colon/dot/% charset pre-check, then a literal parser) would avoid both problems.

3. The broad catch (Exception) in the sync listener is still there

handleAdded / handleUpdated / handleDeleted each wrap the whole body in catch (Exception e) { LOG.error(...) }. I accepted this on 2026-10-21 after confirming the published payload is rebuilt from the DB (selectByDiscoveryHandlerId), so it is observability rather than correctness — a row whose insert failed is simply absent from the published list. But it also means a real DataAccessException on a bulk batch produces only an ERROR line, with no retry, no metric, and no distinction from the expected duplicate-key case. Now that the duplicate-key path is handled by selectByDiscoveryHandlerIdAndUrl + matchByHostAndPort rather than by catching DuplicateKeyException, the catch could reasonably be narrowed to DataAccessException + IllegalArgumentException, or rethrow after logging so the transaction rolls back and the caller sees the failure. Your call, but it should not stay as a blanket Exception.

4. Legacy rows are migrated lazily and there is no upgrade note

normalizeUrl unconditionally appends :80, and rows are rewritten only when an event or admin write happens to touch them; matchByHostAndPort is the compatibility net. Two consequences worth writing into the PR description or RELEASE-NOTES.md (this PR adds no docs and no db/upgrade script):

  • rows stored before this change keep their old form until touched, and the fallback matching is what keeps them working;
  • rows whose stored URL cannot be parsed (unbracketed IPv6, per the contract you settled on) are never matched and never deleted — they stay in discovery_upstream and, since selectByDiscoveryHandlerId returns all rows for the handler, are published to gateways alongside the newly inserted normalized row. A one-shot cleanup statement would close this out.

Nits

  • handleAdded migration branch (else if (!normalizedUrl.equals(existing.getUpstreamUrl()))) only writes upstreamUrl, while handleUpdated copies protocol / upstreamStatus / weight / props onto the matched legacy row. The two migration paths should have the same semantics.
  • Same inconsistency one level down: updateDiscoveryHandlerIdAndUrl sets protocol/upstream_status/weight/props unconditionally, while updateSelective skips nulls. A partial update through the fallback path will null out columns that the primary path leaves alone.
  • final List<DiscoveryUpstreamDO>[] existingListHolder = new List[]{null}; ×3 with @SuppressWarnings("unchecked") — an AtomicReference or a small holder class avoids the unchecked array.
  • checkUrl now hands isHostConnector a bracketed [::1] from the URI branch but an unbracketed ::1 from the parseHostPort branch. InetSocketAddress happens to accept both, so it works, but the two branches should agree on one representation.
  • nativeCreateOrUpdate calls normalizeUrl before the StringUtils.hasLength(id) branch, so the update path now throws IllegalArgumentException on a DTO without upstreamUrl, where updateSelective previously just skipped the column. The only production caller (AbstractShenyuClientRegisterServiceImpl.doDiscoveryLocal) always sets it, so this is defensive only — but the null contract of updateSelective is no longer honoured.
  • parseHostPort uses lastIndexOf(']'); [::1]:8080] yields host ::1]:8080 and fails with a confusing "Invalid IPv6 address". First ] after the opening bracket would give a better message.

What looks good

  • normalizeUrl round-trips are idempotent for IPv4, hostnames and bracketed IPv6, and IpUtilsTest / CommonUpstreamUtilsTest cover the interesting shapes including the trailing-colon (192.168.1.1: → port 80) and missing-bracket cases.
  • validateNumericPort with the 1–65535 range check, and the validateHostIfLooksLikeIP / DIGITS_AND_DOTS pre-filter, are both the right shape.
  • mapToCommonUpstream / mapToDiscoveryUpstreamData returning null instead of throwing, with all four call sites in UpstreamCheckService filtering, and LoadServiceDocEntryImpl dropping its length == 1 ? 80 : ... fallback in favour of the shared parser.
  • checkUrl's http(s) branch is genuinely better than the old split("//")[1] → Integer.parseInt("8080/path") path, which threw NumberFormatException on any URL with a path. The remaining problem is only the null host.

Happy to re-review as soon as the checkUrl null guard is in; #2 and #3 I would like to see addressed in the same push.

@Aias00 Aias00 left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Found a valid IPv6 address form rejected by the new parser. IpUtilsTest#testParseMappedIpv6Address reproduces the failure for [::ffff:192.0.2.1]:8080: IllegalArgumentException: Invalid IPv6 address.

// mismatched platforms.
String hostWithoutZone = host.contains("%") ? host.substring(0, host.indexOf('%')) : host;
InetAddress addr = InetAddress.getByName(hostWithoutZone);
if (!(addr instanceof Inet6Address)) {

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

[P2] Accept IPv4-mapped IPv6 literals during validation

InetAddress.getByName("::ffff:192.0.2.1") returns an Inet4Address on the JDK, even though [::ffff:192.0.2.1]:8080 is a valid bracketed IPv6 endpoint. Requiring the resolved object to be an Inet6Address rejects both dotted and hexadecimal IPv4-mapped literals. Since normalizeUrl() is now used on discovery writes and events, those upstreams are dropped instead of registered.

Please validate the IPv6 literal syntax without relying on the returned InetAddress subclass, or explicitly handle mapped addresses. The added regression test IpUtilsTest#testParseMappedIpv6Address fails with IllegalArgumentException for this input.

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

hold On hold, do not merge

Projects

None yet

Development

Successfully merging this pull request may close these issues.

[BUG] IPv6 upstream addresses are parsed incorrectly by colon splitting

6 participants